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AIR  FORCE  2025  OPERATIONAL 
ANALYSIS 

by  Jack  A.  Jackson ,  Gregory  S.  Parnell 
Brian  L.  Jones ,  Lee  J.  Lehmkuhl 
Harry  W.  Conley  and  John  M.  Andrew 

The  USAF  Chief  of  Staff  determined 
that  the  long-range  planning  process 
within  the  USAF  was  broken.  To  reverse 
this  trend,  he  initiated  a  year-long  study. 
Air  Force  2025,  to  generate  ideas  and  con¬ 
cepts  on  the  capabilities  the  United  States 
will  require  to  possess  the  dominant  air 
and  space  forces  in  the  future,  to  identify 
new  or  high-leverage  concepts  for  employ¬ 
ing  air  and  space  power,  and  to  outline  the 
technologies  required  to  enable  the  envi¬ 
sioned  capabilities.  This  article  outlines  the 
operational  analysis,  which  used  Value- 
Focused  Thinking,  that  supported  the  Air 
Force  2025  study. 


APPRAISING  WARFIGHTING 
CONCEPTS  WITH  WARGAMING 
SIMULATIONS 

by  Robert  M.  Chapman 

Military  operations  research  profes¬ 
sionals  use  analytical  simulations  to  ex¬ 
plore  a  wide  range  of  issues  in  the  acquisi¬ 
tion  and  policy  arenas.  Many  of  these 
analytical  simulations,  however,  are  inap¬ 
propriate  for  examining  warfighting  con¬ 
cepts  because  of  their  simplistic  treatment 
of  human  behavior  especially  decision  pro¬ 
cesses.  In  contrast,  simulations  used  in 
battlestaff  training  programs  are  designed 
to  accommodate  decision  processes.  There 
are,  however,  significant  challenges  in  us¬ 
ing  training  simulations  for  systematically 
examining  military  issues.  This  paper  pre¬ 
sents  a  methodology  for  using  a  training 
simulation  to  explore  a  technology-based 
warfighting  concept.  The  methodology  was 
developed  to  support  JWFC's  participation 
in  the  1996  Defense  Science  Board  Summer 
Study.  Although  developed  and  executed 
in  just  under  two  months,  it  required  a 
significant  commitment  of  resources.  The 
effort,  however,  was  very  productive  be¬ 
cause  it  produced  a  methodology  that  is 
different  from  that  used  in  most  analytical 
simulations-based  studies.  The  methodol¬ 
ogy  used  by  JWFC  blended  techniques 
from  training-based  wargaming  events 
with  those  used  for  analytical  experiments. 
Often  people  assume  there  is  symmetry  be¬ 


tween  simulation-assisted  exercises  and 
analysis  if  they  both  use  computer  models. 
Such  an  assumption,  which  focuses  on  the 
tool  instead  of  the  task,  can  lead  to  flawed 
methodologies.  Understanding  the  differ¬ 
ence,  though,  can  provide  a  useful  tool  for 
military  operations  research. 
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CONCEPT  EXPLORATION  ON  THE 
VIRTUAL  BATTLEFIELD 

by  Gary  Q.  Coe 

1996  Defense  Science  Board  Summer 
Study  on  Tactics  and  Technology  for  21st 
Century  Military  Superiority  addressed  the 
challenge  of  identifying  new  analytic  tools 
for  concepts  evaluation,  motivated  by  the 
evolving  "new  world  order"  that  changed 
missions  and  environments  for  future  war 
and  the  increasing  Department  of  Defense 
investment  in  information  technology  for 
improving  future  warfighting  capability.  In 
support  of  this  effort,  an  experiment  was 
designed  for  evaluating  new  concepts  sug¬ 
gested  by  the  Army  After  Next  and  Marine 
Corps  Sea  Dragon  combat  development 
programs  and  conducted  on  a  future  bat¬ 
tlefield.  The  experiment  was  conducted  at 
the  Institute  for  Defense  Analyses  using 
advanced  distributed  simulation  technol¬ 
ogy  including  equipment  and  personnel 
virtually  interacting  with  a  synthetic  bat- 
tlespace.  It  was  supported  by  the  U.S. 
Army  National  Guard  and  U.S.  Marine 
Corps  active  duty  personnel.  The  Army  Re¬ 
search  Institute  development  human  per¬ 
formance  criteria,  provided  observers  to 
evaluate  human  behavior  of  the  individual 
combatants,  and  assisted  in  data  analysis 
from  the  experiment. 


THE  COST  EXCHANGE  RATIO:  A 
NEW  AGGREGATE  MEASURE  OF 
COST  AND  OPERATIONAL 
EFFECTIVENESS 

by  Bruce  W.  Fowler  and  Pauline  P.  Cason 

Aggregate  measures  of  performance 
have  value  to  analysts  and  decision  makers 
alike  in  comparing  alternatives  and  varia¬ 
tions  on  a  common  scenario.  Traditional 
aggregate  measures  have  included  the 
Force  Exchange  and  Loss  Exchange  Ratios. 
These  measures  are  well  defined  for  single 
system  (homogeneous)  scenarios,  but  re- 
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quire  some  aggregation  methodology  for  a 
multiple  system  (heterogeneous)  scenario.  Dif¬ 
ferent  methodologies  have  been  advanced  to 
perform  this  aggregation  in  a  purely  opera¬ 
tional  context.  An  alternative  aggregation 
methodology  is  cost  of  the  systems  and  their 
ammunition.  This  permits  the  formation  of  a 
Cost  Exchange  Ratio  which  combines  opera¬ 
tional  performance  and  operational  cost  to  pro¬ 
vide  an  aggregate  measure  of  cost  and  opera¬ 
tional  performance.  Combining  this 
methodology  with  an  operational  aggregation 
methodology  permits  definition  of  a  Normed 
Cost  Exchange  Ratio  which  permits  compari¬ 
son  of  the  cost  and  operational  performance  of 
an  alternative  or  variation  force  to  an  equiva¬ 
lent  single  system  base  force,  thus  avoiding  the 
difficulty  of  making  cost  comparisons  across 
cultures.  Together,  these  two  cost  ratios  pro¬ 
vide  an  aggregate  means  for  assessing  cost  vari¬ 
ation  in  an  operational  context,  thus  facilitating 
modern  acquisition  decisions  based  on  such 
considerations  as  Cost  as  an  Independent  Vari¬ 
able. 


AIRBOURNE  AND  SPACE-BOURNE 
RECONNAISSANCE  FORCE  MIXES:  A 
DECISION  ANALYSIS  APPROACH 

by  Terry  A.  Bresnick,  Dennis  M.  Buede, 

Albert  A.  Pisani ,  Leighton  L.  Smith  and 
Buddy  B.  Wood 

Given  the  importance  of  joint  reconnai- 
sance  to  today's  operational  commanders  and 


increasing  reliance  on  reconnaissance  for  the 
future,  the  Joint  Requirements  Oversight  Coun¬ 
cil  (JROC)  recognized  in  1995  that  it  had  no 
means  to  make  force  mix  decisions  in  terms  of 
end-to-end  platform  capability  and  cost  across 
all  components:  manned,  unmanned,  and  over¬ 
head.  To  fulfil  this  need,  the  JROC  created  the 
Reconnaissance  Study  Group  (RSG)  and  tasked 
it  to  develop  and  implement  a  process  for  mak¬ 
ing  timely  and  informed  reconnaissance  force 
mix  decisions.  This  paper  describes  an  innova¬ 
tive  decision  analysis  methodology  for  deter¬ 
mining  the  composition  of  promising  recon¬ 
naissance  architectures  at  various  levels  of 
investment  for  the  2010  time  frame.  The  results 
of  this  study  were  not  developed  to  feed  into 
any  JROC  decision  making  process,  but  to  illus¬ 
trate  a  methodolgy;  nonetheless,  the  results 
have  been  utilized  by  a  number  of  DOD  orga¬ 
nizations. 


Military  Operations  Research ,  V3  N4  1997 


Page  3 


SPECIAL  ISSUE  ON 


MILITARY  OPERATIONS  RESEARCH  METHODS 
FOR  FUTURE  R&D  CONCEPT  EVALUATION 


The  Research  and  Development  (R&D)  funding  decisions  made  by  United  States  military 
planners  will  have  a  large  impact  on  the  technology  base  the  United  States  (and  its  allies) 
will  have  available  to  develop  future  offensive,  defensive  and  information  systems.  The 
purpose  of  these  systems  is  to  deter  or,  when  deterrence  fails,  to  win  armed  conflicts.  In 
addition,  terrorist  bombs,  weapons  of  mass  destruction  and  cyber  warfare  threaten  our 
national  security.  As  defense  spending  has  been  reduced,  less  funding  has  been  available 
for  R&D.  Therefore,  the  R&D  decisions  we  make  in  the  next  few  years  will  be  especially 
critical. 

Limited  R&D  budgets  make  it  imperative  that  we  analyze  the  potential  operational 
benefits  of  future  system  concepts  and  select  the  most  promising  concepts  for  R&D. 
Military  operations  research  offers  several  techniques  to  help  senior  DoD  decision  makers 
prioritize  future  R&D  concepts  and  identify  the  most  promising  technologies  to  pursue. 

The  purpose  of  this  special  issue  is  to  describe  military  operations  research  techniques 
which  have  been  successfully  used  to  evaluate  future  R&D  concepts  and  to  propose 
improvements  to  military  R&D  concept  evaluation  techniques. 


Dr.  Gregory  S.  Parnell 

Editor,  Military  Operations  Research 


ABSTRACT 

n  the  fall  of  1994,  the  Air  Force  Chief  of 
Staff  tasked  Air  University  to  do  a  year¬ 
long  study.  Air  Force  2025,  to  "generate 
ideas  and  concepts  on  the  capabilities  the 
United  States  will  require  to  possess  the 
dominant  air  and  space  forces  in  the  future 
[,  to  identify]  new  or  high-leverage  con¬ 
cepts  for  employing  air  and  space  power  [, 
and  to  detail]  the  technologies  required  to 
enable  the  capabilities  envisioned."  To  sup¬ 
port  this  goal,  we  conducted  an  operational 
analysis  using  Value-Focused  Thinking  to 
identify  high-value  system  concepts  and 
their  enabling  technologies  in  a  way  that 
was  objective,  traceable,  and  robust.  We 
developed  the  Foundations  2025  value 
model  and  identified  the  system  concepts 
that  showed  the  greatest  potential  for  en¬ 
hancing  future  air  and  space  capabilities 
and  the  embedded  technologies  with  the 
highest  leverage  in  making  the  high-value 
system  concepts  a  reality.  Six  alternative 
futures  were  identified  and  a  sensitivity 
analysis  was  performed  to  see  how  sensi¬ 
tive  the  best  systems  were  to  the  alternative 
futures.  The  Foundations  2025  value  model 
was  a  significant  analysis  achievement  and 
has  several  important  future  uses. 

THE  CHALLENGE 

If  we  are  going  to  be  relevant  in  the  future, 
we've  got  to  somehow  break  free  of  the  evolu¬ 
tionary  nature  of  the  planning  process. 

— Gen  Ronald  R.  Fogleman,  CSAF, 
September  6, 1995 

The  Chief  of  Staff  of  the  Air  Force,  Gen 
Ronald  R.  Fogleman,  challenged  the  partic¬ 
ipants  of  the  Air  Force  2025  (AF  2025)  study 
to  generate  ideas  and  concepts  on  the  op¬ 
erational  capabilities  the  United  States  will 
require  to  dominate  air  and  space  forces  in 
the  future  (Fogleman,  1994).  In  response  to 
General  Fogleman's  tasking.  Air  University 
devised  a  four-phase  study  process  (Figure 
1)  to  stimulate  creativity,  generate  ideas, 
and  evaluate  system  concepts  and  technol¬ 
ogies. 

In  the  preparation  phase,  participants 
were  exposed  to  a  wide  variety  of  creative 
thinking  and  problem  solving  concepts. 
This  phase  laid  the  groundwork  for  the 
idea  generation  phase,  in  which  teams  de¬ 
veloped  plausible  alternative  futures  and 
future  system  concepts,  and  identified  the 
enabling  technologies  for  those  concepts. 


During  this  phase,  a  worldwide  data  call 
produced  over  1,000  submissions.  In  the 
assimilation  phase,  the  participants  were 
organized  into  specific  teams  based  on  op¬ 
erational  exposure.  Each  operator  team  fo¬ 
cused  on  a  particular  area,  such  as  interdic¬ 
tion,  information  warfare,  on-orbit  support, 
etc.  After  postulating  the  required  capabil¬ 
ities  of  the  future  Air  Force,  as  part  of  the 
Value  Model  development,  each  team  de¬ 
veloped  system  concepts  and  identified 
technologies  from  the  idea  generation 
phase  that  could  satisfy  these  future  re¬ 
quirements.  This  phase  produced  a  large 
number  of  system  concepts  that  were  de¬ 
scribed  in  varying  levels  of  detail,  provided 
widely  different  kinds  of  operational  capa¬ 
bilities,  and  depended  on  different  levels  of 
advancements  in  different  areas  beyond 
current  technology.  Clearly,  not  all  of  these 
system  concepts  could  be  developed,  nor 
could  all  of  the  technologies  be  aggres¬ 
sively  pursued.  The  study  results  needed  to 
prioritize  the  relative  importance  of  both 
future  system  concepts  and  their  enabling 
technologies  (Fogleman,  1995). 

An  operational  analysis  (OA)  was  con¬ 
ducted  concurrently  with  the  other  three 
phases  to  provide  this  prioritization.  Its 
purpose  was  to  evaluate  system  concepts 
and  the  enabling  technologies  proposed  in 
the  white  papers.  The  operational  analysis 
had  three  objectives: 

•  assess  the  potential  operational  utility  of 
future  air  and  space  system  concepts, 

•  identify  the  high-leverage  technologies 
required  by  those  system  concepts,  and 

•  provide  an  objective,  traceable,  and  ro¬ 
bust  analysis. 

This  paper  highlights  the  main  results  of 
the  operational  analysis;  detailed  docu¬ 
mentation  is  provided  in  the  2025  Opera¬ 
tional  Analysis  Technical  Report  (Jackson, 
et  al,  1996b). 


METHODOLOGY 

After  considering  the  advantages  and 
disadvantages  of  various  analytical  ap¬ 
proaches,  the  OA  team  concluded  that  Val¬ 
ue-Focused  Thinking  (VFT)  (Keeney,  1992) 
offered  the  best  method  for  meeting  the  OA 
objectives  (Jackson,  et  al,  1996a).  VFT  was 
particularly  suited  for  structuring  the  sub¬ 
jective  judgments  required  to  evaluate  the 
systems.  It  also  allowed  the  OA  to  be  com¬ 
pleted  in  the  limited  time  available  and, 
because  VFT  was  used  in  the  SPACECAST 
2020  study  (Burk  and  Parnell,  1996),  it  was 
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Figure  1,  AF  2025  Study  Process 


well  understood  and  accepted  by  the  Air  Uni¬ 
versity  senior  leadership.  In  addition,  once  a 
value  framework  was  built  using  VFT,  it  was 
very  easy  to  assess  systems  across  several  alter¬ 
nate  futures.  Finally,  the  VFT  methodology  en¬ 
ables  the  OA  to  be  objective,  traceable,  and 
robust. 

The  most  important  concept  in  VFT  are  the 
decision  makers  values,  Keeney  (1992)  says, 
"Values  are  what  we  care  about.  [Values] 
should  be  the  driving  force  for  our  decision- 
making."  The  decision  makers  values  are  the 
principles  used  for  evaluation. 

Value  structuring  begins  by  identifying  the 
decision  maker's  values  with  a  hierarchy  of 
objectives.  Top-level  objectives  describe  aspira¬ 
tions  that  are  most  important  to  the  decision 
maker.  A  value  model,  called  a  value  tree  by 
some  authors,  is  a  branching  structure  with  the 
most  fundamental  decision-maker  objectives  at 
the  top.  Keeney  (1992)  uses  the  term  "funda¬ 
mental  objectives  hierarchy,"  and  states,  "The 
higher-level  objective  is  defined  by  the  set  of 
lower-level  objectives  directly  under  it  in  the 
hierarchy."  In  other  words,  the  lower-level  ob¬ 
jectives  completely  specify  their  higher-level 
objective. 

Clemen  (1991)  describes  five  specific  char¬ 
acteristics  of  a  value  model. 

1.  It  should  be  complete,  encompassing  all  im¬ 
portant  facets  of  the  decision. 


2.  It  should  be  as  small  as  possible. 

3.  The  force  qualities  should  allow  straightfor¬ 
ward  measurement. 

4.  Objectives  should  appear  only  once  in  the  tree. 

5.  The  decision  maker  should  be  able  to  think 
about  and  treat  the  branches  of  the  tree  sep¬ 
arately. 

Kirkwood  (1997)  identifies  similar  criteria  for  a 
value  model's  objectives  and  subobjectives.  He 
states  they  should  be  complete,  nonredundant, 
and  independent  to  meet  the  mathematical  re¬ 
quirements  for  a  linear  additive  value  model. 

The  OA  would  contain  three  important  sets 
of  participants  (Figure  2).  The  first  group  was 
the  Air  University  operator  teams  drawn  from 
the  top  20%  of  the  field  grade  ranks  in  the 
military  services.  They  performed  the  role  of 
future  decision-makers  for  this  study;  some 
members  of  this  student  body  will  be  promoted 
to  the  general  officer  ranks  and  will  make  the 
major  acquisition  decisions  for  the  study  time 
frame.  Next,  one  of  the  operator  teams,  the 
Alternate  Futures  team,  was  tasked  with  devel¬ 
oping  a  set  of  plausible  future  scenarios  and 
histories  leading  to  the  year  AF  2025 .  The  third 
group  was  the  Operations  Analysis  team 
tasked  with  performing  the  analysis  for  AF 
2025,  assessing  future  technical  trends,  separat- 
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Figure  2.  AF  2025  Team  Structure 


ing  science  from  science  fiction,  and  evaluating 
the  underlying  high  leverage  technologies. 

Figure  3  provides  a  summary  of  the  oper¬ 
ational  analysis  methodology  we  developed  to 
meet  our  three  study  objectives.  Utilizing  two 
main  drivers,  the  Operators  and  the  Technolo¬ 
gists,  we  constructed  analytical  frameworks  for 
the  Operator  and  Technology  groups.  The  Op¬ 
erators  would  evaluate  the  system  concepts, 
from  the  white  papers,  based  on  the  systems' 
operational  utility  (value).  The  Technologists 
would  evaluate  which  technologies  contributed 
to  the  future  systems.  We  then  ranked  the  sys¬ 
tems  and  technologies  and  concluded  by  per¬ 
forming  sensitivity  analysis. 

The  Search  for  the  AF  2025  Value 
Model 

After  the  OA  team  selected  a  VFT  ap¬ 
proach,  the  next  step  was  to  either  select  an 
existing  value  model  framework  or  develop  a 
new  one.  Identifying  a  current  framework 
proved  to  be  a  daunting  task  because  of  the 
scope  of  the  study  and  the  focus  on  the  far 
future.  Any  potential  value  model  also  had  to 
satisfy  Clemen's  and  Kirkwood's  criteria. 

Time  OA  team  initially  searched  for  a  na¬ 
tional-level  strategic  document  that  identified 
priorities  for  future  air  and  space  forces.  We 
investigated  the  following  sources: 

•  A  National  Security  Strategy  of  Engagement  and 
Enlargement 

•  National  Military  Strategy  of  the  United  States 
of  America  (1995) 

•  Defense  Planning  Guidance 

•  Joint  Requirements  Oversight  Council 
(JROC)/Joint  Warfighting  Capabilities  As¬ 
sessment  (JWCA)  categories 

•  Global  Presence  and  Global  Reach ,  Global  Power 

•  Common  operational  objectives  of  the  armed 
forces  (Kent,  1991) 


•  Draft  Air  Force  Doctrine  Document:  Air 
Force  Basic  Doctrine  (1995) 

•  Joint  Vision  2010  (1996) 

•  Cornerstones  of  Information  Warfare  (1995) 

None  of  the  frameworks  within  these  doc¬ 
uments  met  the  requirements  of  AF  2025  (Par¬ 
nell,  et  al,  1996).  Each  document  was  grounded 
in  current  doctrinal  thinking,  and  the  visionary 
thinking  in  the  white  papers  on  employment  of 
air  and  space  forces  in  the  far  future  did  not 
conform  to  these  frameworks.  Furthermore, 
each  contains  traditional  biases  focusing  on 
how  the  Air  Force  is  organized,  while  AF  2025 
addresses  the  dominant  employment  of  air  and 
space  forces  in  the  year  2025  and  beyond.  The 
only  solution  was  for  the  OA  team  to  develop  a 
new  framework  to  capture  the  visionary  think¬ 
ing  that  took  place  during  the  study.  The  result 
was  the  Foundations  2025  value  model  (Figure 
4).  A  unique  bottom  up  process  was  used  to 
develop  Foundations  2025  (Parnell,  et  al,  1996). 

To  make  the  terms  more  meaningful  to  the 
military  officers,  the  OA  team  used  the  terms 
objective ,  functions ,  tasks ,  and  subtasks  to  desig¬ 
nate  the  tiers  in  the  hierarchy,  from  highest  to 
lowest,  respectively.  Functions  are  the  high- 
level,  aggregated  tasks  that  must  be  accom¬ 
plished  to  attain  the  overarching  objective  of  air 
and  space  dominance.  Three  functions  for  the  fu¬ 
ture  Air  Force  were  identified:  awareness ,  reach, 
and  power .  The  OA  team  adopted  the  following 
definitions  for  these  three  functions: 

Awareness — knowledge,  understanding,  or 
cognizance  of  some  thing  or  situation  through 
alertness  in  observing,  detecting,  and  identify¬ 
ing,  so  as  to  enable,  direct,  and  communicate 
an  informed  decision.  Awareness  is  specified  by 
the  tasks  detect ,  understand ,  and  direct. 

Reach — ability  to  move  to  expand  the  range  or 
scope  of  influence  or  effect,  and  to  sustain  this 
influence  or  effect  by  maintaining  and  replen¬ 
ishing.  To  have  reach  requires  the  ability  to 
deploy,  maintain,  and  replenish. 

Power — ability  to  overtly  or  covertly  affect. 
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Figure  3.  Operational  Analysis  Methodology 


control,  manipulate,  deny,  exploit,  or  destroy 
targets,  including  forces,  people,  equipment, 
and  information,  and  the  ability  to  survive 
while  affecting  targets.  Power  comes  from  the 
ability  to  engage  and  survive. 

The  requirement  for  a  set  of  functions  in  a 
value  model  to  be  mutually  exclusive  and  col¬ 
lectively  exhaustive  results  in  two  critical  im¬ 
plications.  First,  these  three  foundations  2025 
functions  should  encompass  every  future  air 
and  space  force  operational  activity.  Second, 
awareness ,  reach ,  and  power  are  the  only  opera¬ 
tional  activities  that  contribute  to  the  overarch¬ 
ing  objective  of  air  and  space  dominance. 

Figure  5  demonstrates  a  complete  path 
from  the  overall  objective  to  the  bottom  of  a 


branch  of  the  value  hierarchy  ending  with  a 
scoring  function  for  the  measure  of  merit  asso¬ 
ciated  with  communications  capacity.  Follow¬ 
ing  the  outlined  boxes  through  the  hierarchy, 
we  recognize  that  under  the  awareness  func¬ 
tion,  there  are  three  tasks,  detect,  understand, 
and  direct.  The  direct  task  is  composed  of  six 
subtasks,  of  which  communicate  is  a  member. 
The  communicate  subtask  has  five  force  qualities 
capacity.  A  force  quality  defines  a  desired  at¬ 
tribute  of  a  system  to  achieve  a  subtask.  Keeney 
(1992)  states  that,  "[force  qualities]  should  be 
measurable,  operational,  and  understandable." 

Each  force  quality  has  a  measure  of  merit  that 
is  used  to  gauge  system  performance.  In  the 
2025  era,  our  technology  experts  predicted  that 
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Figure  4.  Foundations  2025  Value  Model 


Capacity  (1000  Gigabytes/sec) 

Figure  5.  Value  Framework  Branch  Example 


the  measure  of  merit  for  the  capacity  force  qual¬ 
ity  would  be  how  many  "1000  Gigabytes  per 
second"  could  be  handled  by  a  communica¬ 
tions  channel. 

VFT  scoring  functions  provide  a  quantitative 
means  for  measuring  the  relative  system  per¬ 
formance  for  each  measure  of  merit.  For  Figure 


5  the  corresponding  scoring  function  converts  a 
communications  capacity  into  a  score  from  0  to 
100.  The  example  scoring  function  was  created 
based  on  operational  insights  which  demon¬ 
strate  that  a  curve  with  decreasing  marginal 
returns  accurately  depicts  the  operational  util¬ 
ity  of  this  measure  of  merit. 
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The  value  model  was  developed  following 
the  techniques  recommended  by  Keeney  (1992). 
Tasks  and  subtasks  for  each  function  are  listed 
in  Figure  4,  force  qualities  were  identified  for 
each  subtask,  and  corresponding  measures  of 
merit  were  continually  refined  during  a  succes¬ 
sion  of  meetings.  After  working  with  each  AF 
2025  white  paper  team,  the  OA  team  was  able 
to  reduce  the  list  of  force  qualities  from  the 
initial  number  of  about  1,200  to  the  final  num¬ 
ber  of  134.  The  2025  Operational  Analysis  Tech¬ 
nical  Report  (Jackson,  et  al,  1996b)  contains 
complete  descriptions  of  all  tasks,  subtasks, 
force  qualities,  measures  of  merit,  and  scoring 
functions. 

After  the  hierarchical  structure  of  the  value 
model  is  complete,  the  decision  maker  must 
determine  the  relative  importance  of  the  tiers 
within  the  hierarchy.  This  is  accomplished  by 
the  decision  maker  assigning  numerical 
weights  across  each  tier  of  the  value  model;  for 
a  linear  additive  value  model  these  weights 
must  sum  to  one. 

With  the  development  of  Foundations  2025 
complete,  the  next  step  in  the  AF  2025  opera¬ 
tional  analysis  was  to  use  the  model  to  evaluate 
systems.  The  AF  2025  white  papers  provided 
the  key  information  for  identification  and  def¬ 
inition  of  the  systems. 


System  Concept  Identification 

Following  a  thorough  review  of  the  AF 
2025  white  papers,  the  OA  team  identified  43 
unique  high-leverage  systems  which  fall  into 
the  broad  categories  in  Figure  6.  Also  included 
is  a  brief  description  of  each  system.  For  an 
analysis,  a  system  was  defined  to  be  "a  func¬ 
tionally  related  group  of  elements  that  per¬ 
forms  a  mission  or  task/7  Although  some  of  the 
identified  system  concepts  were  extracted  from 
a  single  white  paper,  many  systems,  particu¬ 
larly  those  involving  the  collection  and  man¬ 
agement  of  information,  were  composites 
drawn  from  capabilities  detailed  in  several  of 
the  papers.  Complete  system  descriptions  are 
contained  in  the  2025  Operational  Analysis 
Technical  Report  (Jackson,  et  al,  1996b). 


Alternate  Futures 

The  AF  2025  Alternate  Futures  team  gener¬ 
ated  and  then  analyzed  over  100  candidate 
drivers  deemed  to  be  forces  acting  on  the  future 
(Engelbrecht  et  al,  1997).  They  selected  the 
three  most  important  drivers  to  define  a  strate¬ 
gic  planning  space  in  which  alternate  futures 
could  be  developed  (Figure  7).  Definitions  for 
each  of  these  three  drivers  are  following. 


Figure  6.  System  Hierarchy 
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Figure  7.  AF  2025  Alternate  Futures  Strategic  Planning  Space 


American  Worldview — This  driver  is  the  US 
perspective  of  the  world  which  determines 
the  nation's  willingness  and  capability  to  in¬ 
teract  with  the  rest  of  the  world.  American 
Worldview  captures  the  dominant  US  focus 
regarding  international  affairs.  The  US  can  be 
primarily  internally  focused,  perhaps  even  iso¬ 
lationist,  or  the  US  can  be  actively  engaged  in 
activities  around  the  world.  The  poles  of 
American  Worldview  are  domestic  and  global. 

A  TeK — This  driver  is  the  differential  in  the 
growth  rate,  proliferation,  leverage,  and  vital¬ 
ity  of  scientific  knowledge  and  technical  appli¬ 
cations  and  their  consequences.  A  TeK  de¬ 
scribes  the  rate  of  change  in  both  the 
proliferation  and  advancement  of  technology. 
The  two  poles  of  A  TeK  are  constrained  and 
exponential.  Constrained  A  TeK  implies  that 
technology  is  advancing  at  an  evolutionary 
rate  and  that  its  availability  is  limited  to  a 
relatively  small  number  of  actors.  Exponential 
A  TeK  occurs  when  there  are  revolutionary 
breakthroughs  in  technology  that  are  rapidly 
proliferated  throughout  the  world. 

World  Power  Grid — This  driver  describes 
the  generation,  transmission,  distribution,  and 
control  of  power  throughout  the  world.  This 


power  is  a  combination  of  economic,  political, 
and  information  sources  of  power  as  well  as 
military  strength.  The  two  poles  of  this  driver 
are  Concentrated  and  Dispersed.  A  concen¬ 
trated  world  power  grid  exists  when  few  ac¬ 
tors  have  the  means  or  will  to  influence  others. 
When  a  myriad  of  groups  or  individuals  can 
change  the  future,  the  world  power  grid  is 
dispersed. 

Six  alternate  futures  were  chosen  from  this 
planning  space  to  provide  a  diverse  set  of  fu¬ 
ture  conditions  used  to  evaluate  the  proposed 
air  and  space  systems.  Four  futures  are  ex¬ 
tremes  and  were  selected  to  bound  the  strategic 
planning  space:  Gulliver's  Travails ,  Zaibatsu , 
Digital  Cacophony,  and  King  Khan.  The  world  of 
Halfs  and  Half-Naughts  was  chosen  for  its  cen¬ 
trality.  Finally,  the  2015  Crossroads  future  pro¬ 
vides  a  conservative  bridge  between  today  and 
2025. 

In  Gulliver's  Travails ,  the  US  is  over¬ 
whelmed  with  worldwide  commitments,  coun¬ 
terterrorism  and  counterproliferation  efforts, 
humanitarian  operations,  and  peacekeeping 
operations.  In  Zaibatsu ,  multinational  corpora¬ 
tions  dominate  international  affairs,  loosely  co- 
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operating  to  create  a  relatively  benign  world. 
Digital  Cacophony  is  the  most  technologically 
advanced  world  resulting  in  great  power  and 
independence  for  the  individual,  but  also  cre¬ 
ating  a  world  of  social  isolation,  fear,  and  anx¬ 
iety.  King  Khan  is  a  world  where  US  dominance 
has  waned  due  to  domestic  problems,  an  eco¬ 
nomic  depression,  and  overshadowing  by  a  ris¬ 
ing  Asian  colossus.  The  world  of  Halfs  and  Half- 
Naughts  is  dominated  by  conflict  between  the 
"haves"  and  "have-nots"  and  by  dynamically 
changing  social  structures  and  security  condi¬ 
tions.  2015  Crossroads  uses  programmed  forces 
from  1996-2001  to  fight  a  major  conflict;  it  pre¬ 
sents  the  US  with  a  strategic  challenge  in  2015 
that  could  lead  to  any  of  other  alternate  futures 
by  2025. 

The  six  alternate  futures  were  used  to  stim¬ 
ulate  the  creativity  of  the  participants  and  to 
assess  six  sets  of  weights  for  the  value  model. 
We  then  were  able  to  do  sensitivity  analysis  to 
find  out  how  the  system  preferences  changed 
with  alternate  futures. 


Weighting  the  Foundations  2025 
Value  Model  Across  Alternate 
Futures 

The  first  step  in  using  the  Foundations  2025 
value  model  is  for  the  decision  maker  to  deter¬ 
mine  the  relative  importance  of  the  functions, 
tasks,  subtasks,  and  force  qualities.  Because  dif¬ 
ferent  futures  dictate  a  different  set  of  required 
air  and  space  capabilities,  the  OA  team  ob¬ 
tained  value  model  weights  from  the  AF  2025 
participants  for  the  six  alternate  futures.  As 
each  alternative  future  was  briefed  to  the  AF 
2025  participants,  the  Operator  teams  were  re¬ 
quired  to  submit  consensus  weights  which  rep¬ 
resented  their  team's  view  of  the  portions  of  the 
value  hierarchy  which  would  have  the  most 
operational  value  in  the  future  scenario.  The 
weights  for  each  future  are  contained  in  the 
2025  Operational  Analysis  Technical  Report 
(Jackson,  et  al,  1996b). 

Each  system  was  scored  against  every  mea¬ 
sure  of  merit  for  each  force  quality.  The  system 
scores  for  each  measure  of  merit  were  weighted 
at  each  level  of  the  hierarchy  by  the  value 
weights.  As  this  process  is  continued — working 
upwards  to  the  top  of  the  value  framework — a 
weighted  average  of  the  system's  scores  across 
the  entire  value  framework  was  developed. 


This  overall  weighted  average  was  the  overall 
system  value. 

Based  on  the  weights  for  any  alternate  fu¬ 
ture,  the  overall  value  of  a  system  concept  can 
be  calculated  by  multiplying  the  score  a  system 
concept  received  for  each  measure  of  merit  by 
the  corresponding  weight  for  the  measure.  The 
overall  value  function  has  the  following  form, 

Overall  Score  System x 

N 

=  'Z  WjWjWkWiWmScoreXn 

n= 1 

where  the  w's  are  the  weights  for  each  level  in 
the  hierarchy;  i  for  function,  j  for  task,  k  for 
sub-task,  1  for  force  quality,  m  for  measure  of 
merit,  and  Score  is  the  value  System  X  received 
on  the  nth  measure  of  merit. 

Use  of  the  linear  additive  value  function 
presupposes  additive  independence  for  each  of 
the  measures  of  merit  (Kirkwood,  1997).  We 
did  not  test  the  134  measures  for  additive  inde¬ 
pendence  because  time  would  not  permit  the 
conduct  of  this  onerous  task.  We  did  test  and 
find  the  top  level  of  the  hierarchy  to  be  mutual 
preferential  independent.  Based  on  this  finding, 
consistent  effort  to  maintain  mutually  exclusive 
development  within  the  hierarchy,  and  recom¬ 
mendations  within  the  literature  to  assume  the 
additive  model  for  rough,  first  cut  analysis,  we 
elected  to  use  the  additive  model. 

RESULTS 

This  section  describes  how  Foundations 
2025  was  used  to  evaluate  future  air  and  space 
systems  and  the  analysis  results. 

Scoring  the  System  Concepts 

A  scoring  team,  selected  from  the  OA  team, 
scored  all  43  systems  against  each  measure  of 
merit  in  Foundations  2025.  A  consensus-seek¬ 
ing  approach  was  used  to  obtain  each  score. 
The  team  was  not  permitted  to  know  the  shape 
of  the  scoring  function. 

The  final  results  of  the  system  scoring  are 
displayed  in  Figure  8.  The  vertical  axis  is  the 
value  from  the  system  evaluation  on  a  scale  of 
0  to  100,  where  a  system  value  of  0  equates  to 
no  value  on  any  of  the  134  scoring  functions. 
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Figure  8.  Final  System  Values 


The  horizontal  axis  is  a  rank  ordering  of  the 
systems  according  to  the  OA  team's  assessment 
of  the  relative  amount  of  technical  challenge  to 
develop  each  system.  Figure  8  shows  system 
values  for  all  six  of  the  alternate  futures.  Each 
system's  values  for  the  various  futures  are  plot¬ 
ted  and  connected  with  a  line  to  show  the  vari¬ 
ation  of  that  system's  value  across  the  alternate 
futures,  and  hence  the  entire  strategic  planning 
space. 

The  curved  dashed  line  provides  a  further 
reference  for  comparing  systems.  In  the  OA 
team's  estimation,  systems  above  the  line  may 
have  sufficient  value  to  offset  the  technical  chal¬ 
lenge  of  producing  such  a  system.  Thus,  sys¬ 
tems  to  the  left  of  the  chart  need  less  value  to  be 
attractive  options  than  systems  to  the  right  of 
the  chart,  because  the  difficulty  of  achieving  the 
capability  is  much  less.  The  location  of  the  line 
is  notional.  It  was  drawn  fairly  low  so  as  not  to 
prematurely  eliminate  any  potentially  promis¬ 
ing  systems  from  consideration. 

the  highest-value  systems  evaluated  in  this 
study  are  the  Global  Information  Management 
System  (GIMS),  Sanctuary  Base,  Global  Area 
Strike  System  (GLASS),  Global  Surveillance 


And  Reconnaissance  System  (GSRT),  and  unin¬ 
habited  combat  air  vehicle  (UCAV).  GIMS  has 
the  highest  value  but  high  technical  challenge; 
GSRT  performs  some  of  the  functions  of  GIMS, 
but  with  less  technical  challenge.  Because  of 
this,  GSRT  could  be  considered  a  "stepping 
stone"  to  GIMS.  Both  GLASS  and  UCAV  score 
well  because  of  a  strong  Awareness  component 
to  complement  their  Power  contributions,  and 
UCAV  is  the  most  feasible  of  all  the  high-value 
systems  in  the  near  term.  The  Sanctuary  Base 
has  high  value  but  also  the  highest  technical 
challenge,  and  may  remain  infeasible  even  be¬ 
yond  2025.  The  2025  Operational  Analysis 
Technical  Report  (Jackson,  et  al,  1996b)  contains 
tables  of  each  system's  value  for  each  future 
and  weight  set. 

The  scoring  results  highlight  the  fact  that  a 
complex  system  (a  system  of  systems)  outper¬ 
forms  any  of  its  components.  This  is  because  of 
the  additive  nature  of  the  scoring  functions. 
The  complex  system  scores  more  broadly  since 
it  contains  the  capabilities  of  all  of  its  compo¬ 
nents.  Conversely,  since  component  systems 
may  have  complementary  capabilities,  the  com- 
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plex  system  will  generally  score  less  than  the 
simple  sum  of  the  component  system  scores. 

Finally,  Figures  9-11  contain  graphs  simi¬ 
lar  to  that  of  Figure  8,  but  for  the  Awareness 
function,  the  Deploy  task  of  Reach,  and  the 
Power  function,  respectively.  These  figures  al¬ 
low  the  reader  to  note  the  systems  that  score 
well  for  a  particular  function.  For  example.  Fig¬ 
ure  9  highlights  the  best  systems  in  terms  of  the 
Awareness  function.  Such  a  level  of  detail  may 
prove  useful  when  conducting  mission  area 
analysis  to  determine  required  improvements 
for  specific  functional  areas.  In  fact,  the  soft¬ 
ware  used  in  this  analysis  can  display  the  sys¬ 
tem  values  at  any  level  of  the  value  model 
(Logical  Decisions™,  1995). 


Scoring  the  Technologies 

Once  the  43  systems  contained  in  the  white 
papers  were  identified,  the  OA  team  qualita¬ 
tively  analyzed  each  system  to  identify  which 
technology  areas  would  be  key  to  achieving  the 
stated  system  capabilities.  Only  those  technol¬ 
ogy  areas  needing  development  were  consid¬ 
ered.  For  example,  if  a  specific  technology  area 
was  critical  to  a  given  system's  capability  but 
no  new  advances  were  needed  in  this  area  for 
the  system  to  achieve  its  full  capability,  then 
this  technology  area  was  not  identified  as  "high 
leverage"  for  this  particular  system. 


The  OA  team  felt  it  highly  desirable  to 
identify  and  group  technologies  according  to  a 
key  baseline  document.  The  Militarily  Critical 
Technologies  List  (1992)  was  used  as  the  basis  for 
identifying  key  technologies  in  each  system 
(Figures  12a  and  12b).  Across  the  43  evaluated 
systems,  43  key  technology  areas  were  identi¬ 
fied  (this  number  is  a  coincidence). 

To  eventually  rank  technologies  by  their 
impact  on  future  air  and  space  capabilities,  the 
team  assigned  a  relative  weight  to  each  tech¬ 
nology  embedded  in  a  particular  system.  The 
weights  selected  add  up  to  100  for  each  system, 
and  so  can  be  thought  of  as  percentages  of  the 
system's  dependence  on  each  technology  need¬ 
ing  development.  For  example,  the  five  piloted 
single-stage-to-orbit  (SSTO)  transatmospheric 
vehicle  (TAV)  technologies  were  weighted  as 
follows: 


Technology  Area  Weight 

High-energy  Propellants  25 

Aerospace  Structures  and  Systems  25 

Ramjet,  Scramjet,  Combined  Cycle  Engines  20 
Advanced  Materials  20 

High-performance  Computing  10 


Once  the  system-versus-technology  matrix 
was  developed,  the  procedure  for  scoring  the 
technologies  was  straightforward.  For  each 
technology,  its  contribution  to  each  system  was 
multiplied  by  the  system  value,  and  the  result- 
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Figure  10.  Deploy  Values  -  "Halts"  Future 


ing  products  were  summed  across  all  systems. 
The  result  was  a  set  of  technology  scores  (nor¬ 
malized  to  a  maximum  score  of  100)  that  takes 
into  account  both  the  technologies'  degree  of 
contribution  to  future  air  and  space  systems 
and  the  importance  of  those  systems  to  air  and 
space  operations.  This  scoring  was  then  re¬ 
peated  for  each  alternate  future  since  the  sys¬ 
tem  value  changed  as  the  weights  changed 
across  alternate  futures.  For  all  six  alternate 
futures,  the  technology  areas  clearly  divided 
into  three  groupings:  the  top  seven  technolo¬ 
gies  (high  leverage),  the  next  five  technologies 
(moderate  leverage),  and  the  bottom  31  tech¬ 
nologies  (less  leverage).  Figure  13  shows  an 
expanded  view  of  the  top  two  technology 
groupings. 

A  common  trend  among  the  higher-lever- 
age  technologies  was  that  they  had  wide  appli¬ 
cability  over  the  systems.  The  high-leverage 
technologies  scored  in  at  least  13  different  sys¬ 
tems;  the  maximum  number  of  systems  where 
any  technology  area  scored  was  27.  Moderate- 
leverage  technologies  scored  in  8  to  12  different 
systems.  High-performance  computing  was  the 
technology  area  with  the  broadest  coverage 
over  the  systems  considered. 


After  each  technology  area  had  been 
scored,  the  technical  assessment  group  assem¬ 
bled  determined  the  key  technology  develop¬ 
ment  leader,  the  DoD  or  the  commercial  sector, 
for  that  particular  area.  They  further  ascer¬ 
tained  the  direction  of  each  developmental  ef¬ 
fort,  whether  from  the  DoD  to  the  commercial 
sector,  from  the  commercial  sector  to  the  DoD, 
or  remaining  constant.  Table  1  summarizes  the 
key  technology  development  leaders  for  the 
high  leverage  technologies:  The  2025  Opera¬ 
tional  Analysis  Technical  Report  provides  this 
data  for  all  technologies  (Jackson,  et  al,  1996b). 


CONCLUSIONS 

The  AF  2025  study  was  publicly  released  by 
General  Fogleman  and  Air  Force  Secretary 
Widnall  in  September  1996  (Air  Force  2025  Fi¬ 
nal  Report  Homepage,  1996).  General  Fogle¬ 
man  stated, 

"Air  Force  2025  demonstrates  to  the  American 
people  that  the  nation's  Air  Force  is  thinking  cre¬ 
atively  and  seriously  about  the  future.  We  brought 
in  a  variety  of  outside  advisors ,  including  some  of 
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the  world's  most  highly  regarded  futurists,  experts 
on  a  wide  variety  of  topics ,  and  forward  looking 
thinkers ,  including  Alvin  Toffler,  Admiral  William 
Owens ,  and  many  others  to  build  a  picture  of  the 
future .  As  part  of  the  comprehensive  long-range 
planning  effort ,  the  alternate  futures  discussed  in 
Air  Force  2025  will  help  develop  a  strategic  vision 
and  prepare  the  United  States  Air  Force  for  the 
challenges  of  the  21st  century " 

— Gen  Ronald  R.  Fogleman,  Air  Force  2025 
Final  Report  Fiomepage,  1996 

The  study  has  been  reported  on  by  the  defense 
press  (Cooper,  1996;  Heronema,  1996;  Fulg- 
hum,  1996;  Tirpak,  1996).  The  study  reports 
covering  all  aspects  of  the  AF  2025  study  can  be 
obtained  on  the  Internet  (Air  Force  2025  Final 
Report  Flomepage,  1996). 

The  AF  2025  Operational  Analysis  was  a 
key  milestone  in  the  AJF  2025  process  and  pro¬ 
vided  a  number  of  unique  contributions.  Most 
importantly,  it  met  its  fundamental  purpose— 
the  OA  identified  future  air  and  space  systems 
required  to  support  air  and  space  dominance 
and  the  key  technologies  that  will  make  those 
systems  possible  by  capturing  the  overall  val¬ 
ues  held  by  the  AF  2025  participants.  Founda¬ 
tions  2025  is  distinguished  by  the  large  number 


of  system  concepts  that  were  analyzed,  the  30- 
year  focus  into  the  future,  and  the  fact  it  was 
developed  through  a  bottom-up  approach. 
Foundations  2025  offers  a  potential  new  frame¬ 
work  for  future  air  and  space  doctrine  that  can 
be  easily  modified  (separated  into  three  mod¬ 
els:  awareness ,  reach,  and  power)  by  Air  Force 
major  commands  for  use  in  their  mission  area 
analysis  process.  Thus,  the  model  presented  is 
an  aid  to  current  and  future  senior  decision 
makers  concerned  with  the  employment  of  air 
and  space  power. 

System  Implications 

The  AF  2025  study  produced  a  number  of 
excellent  system  concepts  for  employing  air 
and  space  power  in  the  future.  This  analysis 
strongly  suggests  that  the  high  ground  of  im¬ 
proved  awareness  offers  significant  potential  for 
achieving  future  air  and  space  dominance.  Typ¬ 
ically,  top-scoring  systems  possessed  higher  de¬ 
grees  of  awareness  and/or  were  predominantly 
space  systems: 

•  Global  Information  Management  System 
(GIMS) 
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Figure  12a.  Technology  Framework 


•  Sanctuary  Base  (SB) 

•  Global  Surveillance,  Reconnaissance,  and 
Targeting  System  (GSRT) 

•  Global  Area  Strike  System  (GLASS) 

•  Uninhabited  Combat  Air  Vehicle  (UCAV) 

•  Space-Based  High-Energy  Laser  (Space  HEL) 

•  Solar  High-Energy  Laser  (Solar  HEL) 

•  Reconnaissance  Unmanned  Air  Vehicle  (Re¬ 
con  UAV) 

•  Attack  Microbots 

•  Piloted  Single-Stage-to-Orbit  (SSTO)  Trans- 
atmospheric  Vehicle  (TAV) 

•  Uninhabited  Air-Launched  TAV 

Seven  of  the  top  eight  systems  emphasized 
the  awareness  function.  GSRT  can  be  thought  of 
as  a  first  generation  GIMS;  it  obtains  most  of 
the  value  of  GIMS  with  much  less  technological 
challenge.  Both  systems  scored  high  because 
the  management  of  information  tasks  was  as¬ 
signed  high  weights  by  the  AF  2025  partici¬ 
pants.  Such  systems  go  beyond  data  fusion  to 
knowledge  fusion;  they  provide  a  global  view 
that  could  revolutionize  military  operations. 
Improved  awareness  is  critically  important  be¬ 
cause  it  enables  virtually  all  other  air  and  space 
force  capabilities. 

Control  of  the  high  ground  of  space  will  be 
very  important.  Of  the  top  11  systems,  only 


three  do  not  operate  in  space  or  use  major 
space-based  components.  Space-based  weap¬ 
ons  are  significant  contributors  to  the  opera¬ 
tional  effectiveness  of  future  air  and  space  op¬ 
erations  by  providing  key  capabilities  in  space 
defense,  ballistic  missile  defense,  defense  of  ter¬ 
restrial  forces,  and  terrestrial  power  projection. 
Of  the  weapon  systems  evaluated,  the  Space 
High  Energy  Laser  (HEL)  seems  to  hold  the 
most  promise,  largely  because  its  optical  sys¬ 
tem  could  also  be  used  for  surveillance  and 
imaging  missions  (an  awareness  function).  Other 
systems  that  scored  well  were  the  Solar  HEL, 
the  Space-Based  Kinetic  Energy  Weapon,  and 
the  Space-Based  High-Powered  Microwave. 
Spacelift  is  another  essential  contributor  to  fu¬ 
ture  space  operations.  Reusable  transatmo- 
spheric  vehicles  provide  critical  lift  capability 
to  improve  virtually  all  space  force  capabilities. 

Improved  power  will  be  best  accomplished 
through  improved  speed,  precision,  and  on- 
station  time.  The  AF  2025  white  paper  teams 
viewed  the  reduction  of  the  OODA  (observe, 
orient,  decide,  act)  loop  to  an  OODA  "point"  as 
critical  to  future  operations.  All  of  the  "shooter" 
systems  that  emphasized  awareness  scored  high 
by  reducing  the  time  to  identify,  target,  and  kill 
threats.  Among  these  systems  are  the  GLASS, 
the  Space  HEL,  and  the  Solar  HEL.  The  envi- 
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Figure  13.  Top  Twelve  Technology  Rankings  (All  43  Systems) 


Table  1.  Technology  Development  Leaders  for  High  Leverage  Technologies 
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sioned  systems  emphasized  the  increased  need  ' 
for  precision  over  mass,  especially  with  respect 
to  avoiding  excess  collateral  damage. 

The  near-real  time  response  requirement  of 
future  combat  meant  many  of  the  systems  either 
were  global  and/ or  pilotless,  such  as  uninhabited 
air  vehicles  (UAVs).  It  is  important  to  note  that 
while  the  UAVs  are  uninhabited,  none  are  envi¬ 
sioned  as  operating  autonomously  without  a  hu¬ 
man  in  the  loop.  Such  an  improved  on-station 
power  capability  is  important  because  it  provides 
a  constant  deterrent  to  enemy  forces. 


Technology  Implications 

The  technology  assessment  portion  of  the 
study  identified  seven  high-leverage  technolo¬ 
gies  that  are  important  to  a  large  number  of 
high-scoring  systems: 

•  Power  Systems 

•  Advanced  Materials 

•  Aerospace  Structures 

•  High-Performance  Computing 

•  Micromechanical  Devices 
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•  High-Energy  Propellants 

•  Data  Fusion 

Advances  in  these  areas  show  promise  to 
substantially  improve  a  wide  range  of  air  and 
space  operations.  Other  technologies  were  also 
important,  but  contributed  to  only  three  or  four 
of  the  high-value  systems.  Among  the  top-scor¬ 
ing  medium-leverage  technologies  were: 

•  Artificial  Intelligence 

•  High-Energy  Laser  Systems 

•  Vehicle  Flight  Controls 

•  Image  Processing 

•  Optics 

Some  of  the  high-leverage  technologies  en¬ 
abling  AF  2025  systems,  such  as  high-perfor¬ 
mance  computing,  are  being  pursued  aggres¬ 
sively  in  the  commercial  sector.  Others,  such  as 
power  systems,  have  lower  commercial  inter¬ 
est.  An  expanded  analysis  of  the  AF  2025  sys¬ 
tems  and  their  embedded  technologies  can  help 
develop  the  most  effective  DoD  investment 
strategy. 

It  is  important  to  remember  that  the  analy¬ 
sis  did  not  take  into  account  the  cost  or  risk  of 
developing  any  of  the  system  concepts.  We 
looked  only  briefly  at  the  technological  chal¬ 
lenge  of  each  system  concept.  While  this  study 
indicates  some  systems  and  technologies  that 
show  promise  for  dramatically  improving  the 
effectiveness  of  air  and  space  operations,  there 
are  other  important  factors  that  need  to  be  con¬ 
sidered  before  making  an  investment  decision. 


Operational  Analysis  Lessons 
Learned 

Foremost  among  the  AF  2025  OA  lessons 
learned  was  that  the  VFT  approach  worked 
very  well.  The  Foundations  2025  value  model 
has  been  used  to  evaluate  systems  that  span  the 
full  range  of  future  air  and  space  combat  oper¬ 
ations.  These  systems  are  conceptual  system 
ideas  that  will  require  significant  research  and 
development  to  design  and  evaluate.  The  OA 
provided  a  structure  to  incorporate  the  subjec¬ 
tive  judgments  of  operational  and  technical  ex¬ 
perts  to  produce  objective,  traceable,  and  ro¬ 
bust  results. 

The  focus  of  the  value  model.  Foundations 
2025,  was  on  the  employment  of  air  and  space 
forces.  This  model  does  not  consider  the  USAF 
functional  areas  required  to  organize,  train,  and 


equip.  By  avoiding  the  current  names  of  mis¬ 
sion  (nouns),  we  developed  a  visionary  value 
model  (using  verbs). 

Major  Implications  for  the  Future 

A  number  of  senior  decision  makers  have 
viewed  the  model  and  commented  that  the  best 
use  of  Foundations  2025  may  be  an  analysis  of 
systems  within  the  distinct  spheres  of  aware¬ 
ness,  reach,  and  power.  They  envision  separating 
and  developing  each  function  of  the  model  fur¬ 
ther  (refining  the  tasks,  subtasks,  force  quali¬ 
ties,  measures  of  merit,  and  scoring  functions) 
and  studying  which  awareness  (or  reach  or 
power)  systems  are  most  promising.  These  three 
separate  models  could  be  effective  mission  area 
analysis  tools  for  the  major  commands. 

The  completed  Foundations  2025  value 
model  is  the  starting  point  for  Value-Focused 
Thinking  with  the  Department  of  Defense.  For 
any  function,  task,  or  subtask,  the  model  can  be 
used  to  evaluate  current  and  projected  systems. 
Next,  the  acquisition  community  can  focus  on 
how  new  concepts  can  be  developed  to  signif¬ 
icantly  increase  value.  Various  creativity  tech¬ 
niques  can  be  used  to  develop  these  new  con¬ 
cepts. 

Another  opportunity  to  capitalize  on  the 
Foundations  2025  model  is  to  use  it  as  a  frame¬ 
work  for  future  air  and  space  doctrine.  Because 
it  identifies  fundamental  functions,  tasks,  and 
subtasks,  it  could  be  the  foundation  for  joint 
doctrine  for  future  air  and  space  warriors.  The 
AF  2025  analysis  techniques  could  be  used  to 
develop  an  entirely  new  joint  military  doctrine 
free  from  current  institutional  bias. 


Summary 

Foundations  2025  represents  five  important 
analytic  advances.  First,  the  collection  of  scor¬ 
ing  functions  serves  as  an  invaluable  resource, 
even  outside  the  AF  2025  study.  Second,  the  use 
of  verbs  to  specify  tasks  was  a  useful  step  in  the 
value  model  evolution.  Third,  building  from 
the  bottom  up  allowed  Foundations  2025  to  be 
free  from  institutional  bias  and  capture  the  vi¬ 
sionary  thinking  of  AF  2025.  Fourth,  Founda¬ 
tions  2025  is  a  very  robust  value  model.  With 
five  tiers  consisting  of  an  overarching  objective, 
three  functions,  eight  tasks,  29  subtasks,  and  134 
force  qualities  (each  with  a  corresponding  mea¬ 
sure  of  merit  and  scoring  function) — and  all 
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weighted  across  six  alternate  futures — the 
model  can  be  used  to  evaluate  very  diverse 
systems.  Finally,  Foundations  2025  is  cast  fur¬ 
ther  into  the  future  than  any  other  known  mil¬ 
itary  value  model. 
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ENDNOTES 

DISCLAIMER.  The  views  expressed  are  those 
of  the  authors  and  do  not  reflect  the  official 
policy  or  position  of  the  United  States  Air 
Force,  Department  of  Defense,  or  the  United 
States  government.  This  paper  contains  fic¬ 
tional  representations  of  future  situations/ sce¬ 
narios.  Any  similarities  to  real  people  or  events, 
other  than  those  specifically  cited,  are  uninten¬ 
tional  and  are  for  purposes  of  illustration  only. 
Mention  of  various  programs  or  technologies 
throughout  this  paper  does  not  imply  Air  Force 
or  DoD  endorsement  of  either  the  mission,  the 
program,  or  adoption  of  the  technology. 
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ABSTRACT 

n  1996,  the  Joint  Warfighting  Center 
(JWFC)  completed  a  wargaming  effort 
for  the  Defense  Science  Board  (DSB). 
JWFC's  six-week  effort  used  a  high-resolu- 
tion  war  game  simulation,  the  Joint  Conflict 
Model,  to  support  the  DSB's  1996  Summer 
Study,  Tactics  and  Technology  for  21st  Cen¬ 
tury  Military  Superiority 2  The  DSB  summer 
study  explored  warfighting  concepts  to  en¬ 
able  "relatively  small  (the  size  to  be  deter¬ 
mined)  and  rapidly  deployable  forces  (or 
teams) — specially  equipped,  trained  and 
supported  by  remote  sensors  and  weap¬ 
ons — to  accomplish  missions  heretofore 
only  possible  with  much  larger  and  massed 
forces/'2 

The  purpose  of  this  paper  is  to  describe 
how  a  wargaming  simulation  normally 
used  for  training  military  leaders  was  used 
to  investigate  a  warfighting  concept.  It  be¬ 
gins  with  a  short  description  of  the  JWFC 
appraisal  approach.  Next  the  DSB  issue  and 
basic  scenarios  are  presented.  The  majority 
of  the  paper  then  describes  how  the  JWFC 
appraisal  team  focused  and  explored  the 
DSB  issue  through  specific  scenario  vi¬ 
gnettes,  definition  of  variables,  and  mea¬ 
surement  of  results.3  Since  the  objective  of 
this  paper  is  to  describe  the  JWFC  appraisal 
methodology,  only  a  sample  portion  of  the 
JWFC  study  results  will  be  presented  and 
the  conclusions  presented  at  the  end  of  this 
paper  will  discuss  the  wargaming  ap¬ 
proach,  not  the  specific  study  results.4  Be¬ 
cause  the  JWFC  effort  was  only  one  seg¬ 
ment  of  the  total  DSB  undertaking,  the 
material  presented  here  is  neither  a  com¬ 
prehensive  review  of  the  entire  DSB  sum¬ 
mer  study  nor  a  critique  of  the  warfighting 
concept. 

THE  JWFC  APPROACH  TO 
WARGAMING  APPRAISALS 

Expertise  in  using  simulations  to  facil¬ 
itate  the  training  of  joint  command  and 
staff  decision  processes  shaped  JWFC's  ap¬ 
praisal  effort.5  The  challenge  in  using  a 
wargaming  simulation  for  the  DSB  study 
was  creating  a  hybrid  methodology  com¬ 
bining  appropriate  elements  of  training 
event  and  analytical  experiment  design. 
Currently,  many  military  training  and  anal¬ 
ysis  activities  use  simulations  to  create  ex¬ 
periments  ("events"  in  joint  training  lexi¬ 


con)  but  usually  differ  in  their  treatment  of 
human  decision  processes.  In  most  simula¬ 
tion-based  analytical  experiments,  human 
decision  processes  are  minimized  or  incor¬ 
porated  into  the  simulation's  algorithms.  In 
contrast,  simulations  used  for  commander 
and  staff  training  events  must  respond  to 
human  decision  processes.  The  JWFC  ap¬ 
praisal  process  borrowed  techniques  from 
both  treatments  and  created  a  simulation- 
based  experiment  with  defined  measure¬ 
ments  and  variables  that  responded  to  hu¬ 
man  decision  processes.  The  variables, 
however,  were  not  changed  incrementally 
as  is  done  in  most  analytical  experiments. 
Instead  they  were  changed  qualitatively  or 
by  an  order  of  magnitude  to  observe  their 
effects.  In  addition,  the  results  of  the  JWFC 
wargaming  experiment  synthesized  nu¬ 
merical  data  and  expert  observations  of  the 
war  gamers.6  (Throughout  the  remainder 
of  this  article,  "wargaming"  will  be  used  to 
describe  an  event  or  experiment  that  cen¬ 
ters  around  mission-related  military  deci¬ 
sion  processes.  "Simulation"  or  "model" 
will  be  used  to  describe  an  inter-active 
computer-based  representation  of  reality.7) 

Conducting  a  wargaming-based  con¬ 
cept  appraisal  effort  is  an  intensive  process. 
Although  JWFC's  DSB  effort  was  rapidly 
accomplished  in  six  weeks,  concept  ap¬ 
praisal  should  normally  be  pursued  at  a 
more  deliberate  pace.  The  nature  of  the 
concept,  complexity  of  the  appraisal  issues, 
and  the  customer's  desired  delivery  date 
would  determine  the  total  time  allocated  to 
the  process.  Between  four  and  six  months  is 
a  reasonable  estimate  for  a  moderately 
complex  concept  appraisal  project.  Figure  1 
depicts  the  process.8 


Design  Stage 

•  Initial  Design  Conference — Determine 
customer  requirements  and  suitability  of 
candidate  issues  to  wargaming-based 
concept  appraisal. 

•  Concept  Refinement — Define  candidate 
issues  as  concepts.  Reduce  concept  to 
essential  attributes  or  functions.  Identify 
those  attributes  and  functions  that  are 
suitable  for  wargaming.  Review  capabil¬ 
ities  and  limitations  of  available  simula¬ 
tion  models  and  other  wargaming  tools. 

•  Initial  Design  of  Experiment — Using  the 
concept's  attributes  and  functions,  de¬ 
scribe  essential  elements  of  appraisal, 
determine  variables  (dependent  and  in¬ 
dependent),  formulate  measures  of  merit 
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Concept  Appraisal  Wargaming  Process 


Total:  4  Months 


Figure  1.  Schematic  of  the  process  used  for  the  JWFC  concept  appraisal. 


and  enabling  scenario  elements.  Select  war¬ 
gaming  tool. 

•  Construct  Test  Plan — Based  upon  the  de¬ 
sign,  develop  the  technical,  wargaming  and 
analytical  support  elements  and  schedule. 

•  Develop  Memoranda  of  Agreement — Obtain 
customer  agreement  on  the  design  and  exe¬ 
cution  of  the  concept  appraisal,  including  the 
content  and  format  of  the  final  concept  ap¬ 
praisal  product. 

Preparation  Stage 

•  Produce  Detailed  Scenario — Create  the  back¬ 
ground,  assumption  and  operational  topog¬ 
raphy. 

•  Develop  Detailed  Wargaming  Plan — De¬ 
scribe  concept  appraisal  attributes  and  func¬ 
tions  in  terms  of  model  events  or  processes. 

•  Develop  Detailed  Data  Collection  Plan — 
Determine  data  elements  and  source. 


•  Build  Database — Create  new  or  modify  ex¬ 
isting  database. 

•  Simulation  and  Database  Verification — 
Conduct  functionality  tests  of  the  model  and 
database  based  on  how  the  warfighting  con¬ 
cept  is  to  be  implemented  in  the  simulation. 
Conduct  model  proficiency  training  for  war 
gamers  if  required.9 

Wargaming  Stage 

•  Validation  Runs — Run  representative  sam¬ 
ple  of  wargaming  events.  Practice  data  col¬ 
lection  process.  Check  results  for  reasonable¬ 
ness,  consistency,  and  expected  outcomes. 
Determine  if  modifications  to  test  design  or 
plan  are  needed.10 

•  Conduct  Final  Runs — Run  all  wargaming 
events  and  collect  data.  Evaluate  each  run  for 
acceptability  and  data  integrity.  Repeat 
events  that  are  unacceptable. 
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Appraisal  Stage 

•  Data  Reduction — Manipulate  data  into  a  for¬ 
mat  suitable  for  appraisal. 

•  Data  Appraisal — Synthesize  quantitative 
and  observer-based  data. 

•  Prepare  Concept  Appraisal  Results — Pro¬ 
duce  and  deliver  final  product  to  customer. 
The  remainder  of  this  paper  will  discuss  a 
few  of  elements  of  this  process. 

CONCEPT  REFINEMENT— THE  DSB 
STUDY  ISSUE  AND  SCENARIOS 

The  DSB  Summer  Study  issue  was:  Can 
improved  technologies  enable  small  US  land 
forces  to  perform  missions  previously  accom¬ 
plished  by  larger  forces?  The  enabling  concept 
was  a  combination  of  tactics  and  technology. 
Advanced  sensors  would  be  used  to  produce 
and  distribute  battlefield  information  that 
would  provide  local  situation  awareness  to 
small  land  forces.  These  units  would  then  rely 
on  long-range  indirect  fires,  such  as  arsenal 
ships  or  armed  aerial  vehicles,  to  engage  adver¬ 
saries  while  avoiding  direct  engagement.  To  be 
feasible,  from  both  effectiveness  and  survivabil¬ 
ity  perspectives,  the  timelines  between  the  ac¬ 
quisition  and  engagement  of  targets  must  be 
very  short.11  The  DSB  Summer  Study  postu¬ 
lated  seven  scenarios  for  exploring  the  concept: 

•  Halting  a  combined  arms  attack  (Desert 
Storm). 

•  Securing  territory  (Bosnia). 

•  Urban  warfare  against  a  CW/BW  facility. 

•  Extended  offensive  operations  to  attack  en¬ 
emy  forces  in  an  occupied  nation. 

•  Extraction  of  a  brigade-size  force. 

•  Sea  Dragon  (USMC  FOFAC  concepts). 

•  Neutralizing  a  deeply  buried,  hardened  facil¬ 
ity. 

JWFC  was  brought  into  the  effort  in  early  sum¬ 
mer  with  six  weeks  to  produce  appraisal  re¬ 
sults.  Because  of  this  time  constraint,  JWFC  ex¬ 
amined  only  one  scenario,  extended  offensive 
operations  to  attack  enemy  forces  in  an  occu¬ 
pied  nation. 

The  scenarios  provided  by  the  DSB,  how¬ 
ever,  lacked  detail.  It  was  left  to  the  participat¬ 
ing  teams  to  craft  and  articulate  the  relevant 
elements  of  each  scenario.  The  JWFC  team  be¬ 


gan  creating  these  elements  by  laying  the  stra¬ 
tegic-  and  operational-level  foundations  of  the 
of  the  tactical-level  wargaming  scenario.12  This 
step  was  especially  important  because  at  an 
initial  DSB  summer  study  conference  some 
guest  speakers  questioned  the  direction  of  the 
DSB  study.  They  maintained,  for  example,  that 
the  strategic  and  operational  prerequisites  for 
the  DSB  warfighting  concept,  such  as  domi¬ 
nance  of  the  maritime  and  aerospace  environ¬ 
ments,  would  make  the  DSB  warfighting  con¬ 
cept  irrelevant.  Why  put  small,  light  land  forces 
at  risk  when  the  targets  could  be  attacked  by 
naval  and  air  systems?  It  was  critical,  therefore, 
for  the  JWFC  team  to  construct  the  strategic- 
and  operational-level  dimensions  of  the  se¬ 
lected  scenario  that  would  support  credible  tac¬ 
tical-level  exploration  of  the  DSB  warfighting 
concept. 

SCENARIO  DEVELOPMENT 

The  creation  of  the  strategic  and  opera¬ 
tional-level  dimensions  of  the  scenario  mir¬ 
rored  the  process  used  by  JWFC  to  develop 
scenarios  and  adversary  campaign  plans  for 
major  exercise  events.  This  process  produced 
the  assumptions,  initial  conditions,  and  adver¬ 
sary  force  characteristics  that  would  logically 
frame  the  pursuit  of  tactical-level  missions. 
Alaskan  terrain  was  used  for  its  geographical 
diversity,  but  the  geopolitical  landscape  was 
fictional.  Acadia,  the  adversary  force,  invaded  a 
province  of  Aleutia,  a  nation  whose  territorial 
integrity  was  vital  to  US  national  interests.  The 
initial  US  response  employed  US  naval  and  air 
forces  to  attack  the  invading  Acadian  forces. 
The  simulation  playbox  was  a  portion  of  this 
scenario,  the  area  of  operations  for  a  heavy 
Acadian  corps.  The  corps  was  reduced  in 
strength  by  US  air  and  naval  action  to  division- 
level  strength.  The  Acadian  corps  dispersed  its 
remaining  units  both  to  reduce  its  vulnerability 
to  air  and  naval  attacks  and  to  establish  control 
of  the  Aleutian  province.  In  the  JWFC  study 
scenario,  the  Acadian  forces  were  equipped 
with  current  US  weapons  systems  and  there 
were  no  viable  Aleutian  forces  remaining  in  the 
area  to  support  US  operations. 

Another  critical  portion  of  the  scenario  was 
the  design  of  the  US  ground  forces  that  were  to 
use  the  DSB's  warfighting  concept.  Here  again, 
the  DSB  concept  lacked  the  details  necessary  to 
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model  and  employ  these  forces.  In  a  wargam¬ 
ing  simulation,  units  and  systems  must  be 
modeled  with  specific  parameters:  number  of 
personnel,  movement  rates,  sensor  acquisition 
characteristics,  weapons  range,  lethality,  etc.  In 
addition,  the  war  gamer  must  employ  the  units 
using  the  appropriate  doctrine — a  mix  of  tac¬ 
tics,  techniques  and  procedures  (TTPs)  unique 
to  each  type  of  unit.  While  a  new  technology 
may  be  exciting  and  potentially  useful,  in  the 
end  it  must  be  coupled  to  appropriate  military 
doctrine  and  missions.  When  the  forces  are  a 
theoretical  construct  of  the  future,  the  war 
gamer  must  extrapolate  current  doctrine  into 
the  future  and  derive  TTPs  appropriate  to  the 
imaginary  units'  missions  and  capability.  Unit 
organization,  size,  and  employment  tactics 
must  accommodate  the  weapons  and  systems 
derived  from  the  concept.  In  the  case  of  radi¬ 
cally  new  concepts,  it  is  difficult  to  justify  the 
mere  patching  of  new  weapons  and  systems 
onto  existing  formations  and  doctrine.  These 
issues  were  also  discussed  at  a  DSB  conference. 
The  JWFC  study  team's  solution  was  the  cre¬ 
ation  of  a  new  kind  of  unit.  Brigade  2015. 

Brigade  2015  was  conceived  as  a  strategi¬ 
cally  mobile  and  technologically  advanced 
ground  force.  In  this  scenario,  it  was  the 
ground-based  exploitation  element  of  the  initial 
US  air  and  naval  response.  The  brigade  was 
inserted  into  Aleutia  by  a  mix  of  strategic  and 
theater  air  assets.  In  Aleutia,  its  mission  was 
control  of  key  terrain  and  facilities  and  to  attack 
Acadian  forces  that  were  mobile,  perishable, 
and  dispersed — targets  not  suitable  for  engage¬ 
ment  by  autonomously  operating  air  and  naval 
weapons  systems.  Brigade  2015  was  dispersed 
into  numerous  teams  within  its  area  of  opera¬ 
tion.  Each  team,  the  brigade's  basic  combat  el¬ 
ement,  was  composed  of  10  personnel  and 
could  move  by  foot  or  helicopter.  The  teams 
had  immediate  access  to  a  theater-level  tier  of 
distributed  information  that  provided  sufficient 
intelligence  for  situation  awareness  but  not  de¬ 
tailed  enough  for  targeting.  For  weapons  tar¬ 
geting,  each  team  relied  on  another  set  of  lo¬ 
cally  controlled  systems  either  organic  to  the 
team  or  provided  by  a  team  Unmanned  Aerial 
Vehicle  (UAV).  Although  the  teams  had  some 
organic  fire  support  capability,  they  avoided 
direct  engagement  with  Acadian  forces.  In¬ 
stead,  they  relied  on  long-range  air  support, 
arsenal  ships  or  a  UAV  with  a  directed  energy 
weapon  to  engage  Acadian  forces.  Brigade  mis¬ 
sions  would  be  limited  initially  to  synchronized 


missions  by  a  few  of  the  teams.  As  the  situation 
matured,  the  brigade  would  bring  together  its 
teams  and  transition  to  battalion  and  brigade¬ 
sized  missions. 

The  scenario  elements  were  then  translated 
into  Joint  Conflict  Model  (JCM)  database  ele¬ 
ments.  JCM  is  a  high-resolution  training  simu¬ 
lation  that  depicts  military  activities  on  a  map¬ 
like  video  display.  The  simulation  war  gamer 
controls  forces  through  orders  and  is  presented 
information  based  upon  the  capabilities  and 
actions  of  his  assigned  forces.  Figure  2  summa¬ 
rizes  the  general  scenario  elements  and  dis¬ 
plays  the  JCM  simulation  playbox  (a  portion  of 
occupied  Aleutia)  with  Acadian  adversary 
forces. 


SCENARIO  VIGNETTES 

Initially  the  JWFC  team  envisioned  a  single 
extensive  war  game  scenario  that  included  de¬ 
ployment,  employment  and  sustainment  of  the 
brigade.  During  the  design  stage,  the  scope  was 
quickly  reduced  because  the  JWFC  team  found 
that  effort  diverted  to  a  broad  treatment  began 
to  obscure  the  essential  elements  of  the  issue. 
Focus  was  restored  by  developing  several  em¬ 
ployment  missions  that  would  be  plausible 
missions  for  Brigade  2015  to  undertake  within 
the  general  scenario.  Deployment  and  sustain¬ 
ment  of  the  brigade,  while  obviously  impor¬ 
tant,  was  dropped  from  the  experiment.  The 
team  felt  that  the  first  critical  test  of  the  DSB 
concept  was  employment.  Investigating  the  de¬ 
ployment  and  sustainment  aspects  of  the  con¬ 
cept  would  be  necessary  only  if  the  concept 
could  accomplish  employment  missions. 

Eleven  brigade  team  missions  were  devel¬ 
oped  that  encompassed  a  spectrum  of  missions 
likely  to  occur  within  the  general  scenario.  Af¬ 
ter  examining  the  characteristics  of  each  mis¬ 
sion,  the  team  selected  four  that  effectively 
sampled  all  the  characteristics  present  in  the 
original  eleven.  The  four  missions  were  then 
fully  developed  into  scenario  vignettes.  The  se¬ 
lection  process  was  similar  to  that  used  in  de¬ 
signing  analytical  experiments  around  an  antic¬ 
ipated  response  surface.  Each  vignette  and  the 
embedded  mission  characteristics  were  suffi¬ 
ciently  diverse  to  give  the  JWFG  team  confi¬ 
dence  that  they  would  explore  a  representative 
range  of  environments  and  potential  outcomes. 
The  four  vignettes,  depicted  in  relation  to  the 
general  scenario,  are  displayed  in  Figure  3. 
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General  Scenario  -  Invasion  of 
Aleutia  by  Acadia 

•  Invasion  force  consisted  of  a  heavy  corps  ground  force 
supported  by  fixed  and  rotary-wing  air  assets. 

•  US  air  and  naval  units  conduct  strategic  and 
operational-level  attacks.  Air  and  maritime  dominance 
established. 

•  Acadian  force,  dispersed  throughout  Aleutia,  is 
consolidating  its  control  and  occupation. 

Specific  Scenario  -  Brigade  2015,  a 

strategically  mobile,  technologically 

advanced  ground  force  inserted  into 
Aleutia. 

•  First  land  force  follow-on  to  initial  US  air  and  naval 
responses 

•  Brigade  dispersed  throughout  area  of  operations,  is 
organized  around  small  teams  capable  of  either 
independent  or  synchronized  operations. 

•  Brigade  teams  attack  Acadian  forces  that  are  mobile, 
perishable  and  dispersed  -  targets  not  suitable  for 
autonomously  operating  air  and  naval  weapons. 

•  Teams  avoid  engagement  with  Acadian  forces. 

Instead  they  locate  and  target  Acadian  forces  for 
attack  by  long-range  supporting  fires  and  air  support 


Scenario  map  of  Acadian 
occupation  forces  -  the  “playbox” 
for  the  war  game. 


Towns 


v  Roads 


Forest 


Water 


* 


a 


Figure  2.  This  schematic  depicts  the  scenario  used  for  the  JWFC  concept  appraisal.  The  map  shows  the 
Acadian  forces  remaining  in  a  sector  of  Aleutia  after  US  naval  and  air  attacks.  There  are  no  friendly  Aleutian 
forces  remaining  in  this  sector.  These  forces  comprised  the  adversary  for  the  war  game  scenario  vignettes. 


VARIABLES 

Another  challenge  was  defining  and  mea¬ 
suring  test  variables.  The  JWFC  study  team 
spent  a  considerable  amount  of  time  wrestling 
with  the  problem  of  how  and  what  to  measure. 
One  source  of  frustration  was  the  attempt  to 
use  measurements  that  were  very  compelling 
from  an  analytical  perspective  but  which  were 
inappropriate  to  a  wargaming  environment. 
Another  distraction  was  the  consideration  of 
measurements  so  detailed  that  they  obscured 
the  proposed  warfighting  concept.13  Finally, 
the  study  team  recognized  that  in  wargaming, 
like  many  other  undertakings,  simplicity  in¬ 
vokes  clarity.  Several  dozen  candidate  mea¬ 
sures  were  reduced  to  six  measurements:  four 
independent  variables  and  two  dependent  vari¬ 
ables. 

At  the  same  time,  the  JWFC  team  consid¬ 
ered  the  methodology  for  manipulating  the  in¬ 
dependent  variables.  Clearly,  it  would  not  be 
feasible  to  examine  a  large  number  of  potential 


values.  Yet,  there  was  a  desire  to  observe  the 
response  of  the  dependent  variables  to  changes 
in  independent  variables.  Typically,  in  sensitiv¬ 
ity  analyses,  independent  variables  are  allowed 
to  vary  nearly  continuously  but  within  a  re¬ 
stricted,  marginal  domain.  The  JWFC  appraisal, 
in  contrast,  allowed  the  independent  variables 
to  take  on  only  one  of  two  values.  The  domain, 
however,  was  very  wide,  representing  order- 
of-magnitude  changes  in  the  variables.  The  four 
independent  variables  and  their  allowable  do¬ 
main  values  were: 

•  Mobility  (how  fast  the  teams  could  move) — 
foot  mobile  or  helicopter  transport. 

•  Sensors  (the  ability  of  the  teams  to  acquire 
and  track  adversary  forces) — basic  (unit  and 
weapons  systems)  or  basic  plus  a  reconnais¬ 
sance  UAV. 

•  Long  Range  Fire  Support  (fire  support  assets 
available  in  addition  to  the  teams'  organic 
weapons) — four  F/A-18  sorties  and  an  arse- 
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Four  Vignettes  -  snapshots  of 
brigade  operation  that  provide  range 
of  plausible  mission  environments 
and  the  context  for  examining 
warfighting  concepts 

•  Defend  Mining  Area  -  9  teams  defend  mineral 
mining  area  from  Acadian  attack 

•  Scud  Hunt  -  3  teams  locate  and  destroy  Acadian 
missile  brigade 

•  Area  Ambush  -  9  teams  ambush  Acadian  convoy 

•  Meeting  Engagement  -9  teams  inadvertently  contact 
Acadian  battalion 

Independent  Variables 

•  Mobility  -  infantry  or  helibome 

•  Sensors  -  basic  (unit  and  weapons)  or  basic  plus 
teamUAVs 

•  Supporting  Fires  -  4  F/A-18  sorties  and  arsenal  ship 
or  armed  UAV 

•  Sensor-shooter  time  -  short  (1  min.)  or  long  (11  min.) 

Dependent  Variables 

•  Mission  effectiveness 

•  Unit  survivability 


Location  of  mission  vignettes 
within  the  general  scenario. 


<4 - 220  Km - — ►  ± 


Figure  3.  This  schematic  depicts  the  spatial  relationship  among  the  four  scenarios.  The  four  scenarios  were 
treated  independently — the  outcomes  in  one  vignette  did  not  impact  either  side's  behavior  or  resources  in 
another  vignette.  Each  vignette  was  wargamed  several  times  using  different  values  of  the  independent 
variables. 


nal  ship  or  four  F/A-18  sorties  and  an  armed 
UAV  with  a  directed-energy  weapon. 

•  Sensor-to-Shooter  time  (length  of  time 
needed  to  call  for  fire  support  after  a  sensor 
acquired  a  target) — 11  minutes  or  1  minute 
(Note:  Sensor-to-Shooter  time  only  applied 
to  on-call  fire  support;  pre-planned  fire  sup¬ 
port  occurred  at  the  designated  time.) 

The  two  dependent  variables  and  their  al¬ 
lowable  ranges  were: 

•  Mission  effectiveness — adversary  personnel 
and  systems  killed  or  destroyed,  normalized 
between  0  (none)  to  100  (all). 

•  Unit  survivability — brigade  team  personnel 
and  systems  remaining,  normalized  between 
0  (none)  to  100  (all). 

With  four  vignettes  and  four  independent 
variables  capable  of  exhibiting  two  values  each, 
there  are  64  unique  combinations  suitable  for 


wargaming.  The  JWFC  study  team  had  neither 
the  resources  nor  time  to  run  each  unique  com¬ 
bination  and  reduced  the  test  schedule  to  26 
runs.  Some  reduction  was  possible  by  analyz¬ 
ing  the  dependent  variable  characteristics.  For 
example,  the  armed  UAV  had  its  own  acquisi¬ 
tion  and  tracking  systems;  therefore,  it  was  sel¬ 
dom  used  in  conjunction  with  a  reconnaissance 
UAV.  Also,  because  of  the  characteristics  of  the 
armed  UAV,  Sensor-to-Shooter  time  was  al¬ 
ways  1  minute  if  a  brigade  team  had  an  armed 
UAV.  The  Sensor-to-Shooter  varied  only  when 
the  arsenal  ship  was  used.  Mobility  varied  only 
in  the  fourth  vignette.  The  resulting  test  sched¬ 
ule  is  displayed  below: 

•  Vignette  1:  2  runs.  Sensors  varied. 

•  Vignette  2: 6  rims.  Sensors,  Long  Range  Fire 
Support,  and  Sensor-to-Shooter  Time  varied. 

•  Vignette  3:  6  runs.  Sensors,  Long  Range  Fire 
Support,  and  Sensor-to-Shooter  Time  varied. 
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•  Vignette  4:  12  runs,  6  runs.  Sensors,  Long 
Range  Fire  Support,  Sensor-to-Shooter  Time, 
and  Mobility  varied. 

In  addition  to  the  two  quantitative  depen¬ 
dent  variables,  a  qualitative  measure  of  out¬ 
come  was  also  developed.  For  each  scenario 
vignette,  an  adversary  and  brigade  command¬ 
er's  mission  statement  was  created.  The  mis¬ 
sion  statements  provided  a  tactical  plan  for 
both  adversary  and  brigade  war  gamers  and 
described  the  objective  of  each  side  in  a  specific 
vignette.  More  importantly,  though,  it  defined 
mission  success  for  each  side.  In  hindsight,  cre¬ 
ation  of  a  commander's  mission  statement  was 
the  most  important  element  in  the  scenario  de¬ 
sign  process.  It  gave  purpose  and  discipline  to 
each  vignette  by  allowing  the  war  gamers  to 
plan  and  act  as  tactical  unit  commanders.  Dur¬ 
ing  the  test  runs,  while  the  simulation  was  re¬ 
cording  the  numerical  data,  the  war  gamers 
were  recording  their  personal  observations. 
Blending  quantitative  data  and  qualitative  ob¬ 
servations  into  one  coherent  account  would  not 


have  been  possible  without  the  structure  im¬ 
posed  by  a  mission  statement.  At  test  comple¬ 
tion,  the  attainment  of  each  side's  mission  state¬ 
ment  was  found  to  give  the  clearest  measure  of 
outcome,  better  than  the  quantitative  measures 
of  mission  effectiveness  or  unit  survivability. 

Figures  4  through  7  display  each  of  the  four 
vignettes.  Each  figure  contains  an  abbreviated 
description  of  the  vignette,  a  tactical  plan  over¬ 
laid  on  the  JCM  graphic  map,  and  a  table  which 
lists  the  run  identification  number  with  the  in¬ 
dependent  variables. 


EXAMPLES  OF  RESULTS 

As  previously  described,  test  results  were 
synthesized  from  quantitative  data  and  quali¬ 
tative  observations.  As  enumerated  in  Table  1, 
the  observations  of  the  war  gamers  in  relation 
to  the  accomplishment  of  the  commander's 
mission  statement  were: 


Acadian  battalion-size 
force  attacks  Aleutian 
mining  area  defended  by 
company-size  force  of 
brigade  teams. 

*  Acadian  mission  is  controlling  terrain 

•  Brigade  mission  is  denying  Acadian 
access  to  area 


Run 

Mobility 

Infantry 

X_ 

X 

Helicopter 

Sensors 

KH 

Standard 

X 

X  I 

TeamUAV 

X~1 

Long  Range  Fires 

Air  Support 

■■ 

F/A-18  (4  sorties) 

2L_ 

X 

Arsenal  Ship 

X 

X 

Armed  UAV 

Sensor-shooter  Time 

1  Minute 

X 

X 

|  11  Minutes 

_ 

_ 

IS  Brigade  Teams 

Brigade  Controlled 
Obstacle  (minefield) 


Acadian  Mechanized 
Infantry  Battalion 

Acadian  Maneuver 
Axis 


Vignette  1 


”35  Km" 


Aleutian 
Mining  Area 


Roads 


>^X  Towns 


Forest 


Figure  4.  Vignette  1. 
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Vignette  2 


Run _ 

Mobility _ 

Infantry _ 

Helicopter _ 

Sensors _ 

Standard 

TeamUAY _ 

Long  Range  Fires 

Air  Support _ 

F/A-18  (4  sorties) 
Arsenal  Ship 

Armed  UAV _ 

Sensor-shooter  Time 

I  Minute 

I I  Minutes 


3  15  17  19  1 11  1 1 


X 


X 


X 


X 


JSI 


Brigade  Teams 


Missile  Launcher  Site 
Acadian  Security  Elements 
(Air  Defense,  Infantry,  Artillery) 


Roads 


Towns 


Forest 


Platoon-size  force  of 
brigade  teams  locates  and 
attacks  a  dispersed 
Acadian  missile  brigade. 

♦  Brigade  mission  is  reconnaissance  and 
indirect  fire  engagement  of  missile 
launchers. 

•  Acadian  mission  is  area  security. 


Figure  5.  Vignette  2. 


•  For  the  26  runs ,  brigade  teams  won  12  and 
adversary  forces  won  11  (based  on  mission 
statement;  3  were  draws  (neither  accom¬ 
plished  mission). 

•  Brigade  teams  always  won  if  they  had  use  of 
the  Armed  UAV. 

•  Brigade  teams  never  won  if  they  had  only  the 
arsenal  ship  and  standard  sensors . 

•  Brigade  teams  won  4  and  lost  2  (3  draws)  if 
they  had  an  arsenal  ship  and  team  UAV 
sensors. 

Quantitative  results  were  presented  in 
three  formats.  The  first  consisted  of  comparing 
quantitative  measures  of  mission  effectiveness 
and  unit  survivability  within  a  specific  vi¬ 
gnette.  Figure  8  displays  one  of  these  formats 
from  Vignette  4.  Note  that  the  quantitative 
measures  are  generally  consistent  with  the 
qualitative  observations  of  mission  accomplish¬ 
ment  displayed  above. 

Another  format  for  displaying  data  was 
comparing  changes  in  an  independent  variable 
to  the  two  dependent  variables  within  like  pairs 
of  runs.  For  example,  runs  3  and  5  were  alike 


except  for  the  value  of  the  fire  support  variable. 
Run  3  used  the  armed  UAV  and  run  5  had  the 
arsenal  ship  (as  explained  previously,  both  also 
used  four  F/A-18  sorties).  This  was  done  for 
each  independent  variable,  resulting  in  8  total 
charts.  Figures  9  and  10  display  two  of  these 
charts,  the  data  for  the  fire  support  variable 
measured  against  mission  effectiveness  and 
unit  survivability  between  like-run  pairs. 

The  variable-oriented  data  was  also  consis¬ 
tent  with  expected  results,  showing  a  positive 
relationship  between  "better"  values  of  the  in¬ 
dependent  variable  and  outcomes  favorable  to 
the  US  side.  However,  the  value  of  this  format 
was  not  in  showing  the  relative  sensitivity  of  an 
independent  variable  to  outcomes  in  a  specific 
mission  category  or  vignette.  Rather,  it  pro¬ 
vided  a  confidence  check  of  individual  rims 
and  the  general  methodology.  Apparent  anom¬ 
alies  were  examined  to  determine  their  causes. 
For  example,  in  the  9/11  run  pair  (Figure  9),  the 
mission  effectiveness  measure  went  up  when 
the  arsenal  ship  was  used  instead  of  an  armed 
UAV.  This  data  was  incongruent  with  other  run 
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Company-sized  force  of 
brigade  teams  attacks  an 
|  Acadian  battalion  moving  along 
a  highway. 

*  Brigade  mission  is  to  ambush  moving  Acadian 
forces 

►  Acadian  mission  is  conducting  a  secure 
movement  of  a  convoy. 


Vignette  3 

30  Km- 


Brigade  Teams 


Brigade  Controlled 
Obstacle  (minefield) 


Acadian  Support  Elements 
(Air  Defense,  Rocket  Artillery, 
Artillery) 


Acadian  Convoy 
(Transportation, 
Armored  Calvary  units) 


Figure  6.  Vignette  3. 


pairs.  Examination  of  the  vignette  observations 
from  the  war  gamers,  however,  quickly  re¬ 
vealed  an  explanation.  In  run  9  the  brigade 
teams  were  able  to  quickly  accomplish  their 
mission,  destruction  of  three  Scud  launchers, 
with  minimal  engagement  of  the  Acadian  unit's 
security  forces.  In  run  11,  however,  the  brigade 
teams  only  destroyed  two  of  three  launchers 
but  inflicted  a  considerable  amount  of  collateral 
damage  to  other  elements  of  the  Acadian  mis¬ 
sile  unit.  Since  the  quantitative  measure  of  mis¬ 
sion  effectiveness  counted  all  elements  of  the 
Acadian  unit,  the  measure  was  higher  even 
though  the  war  gamers  graded  run  11  as  a 
draw — neither  side  met  its  mission. 


CONCLUDING  OBSERVATIONS 
ABOUT  WARGAMING 

The  JWFC  study  team  knew  that  they  were 
employing  a  hybrid  approach  by  combining 


techniques  from  the  training  and  analytical  do¬ 
main  but  when  combined,  might  exhibit  unique 
characteristics  and  challenges.  While  some  of 
these  characteristics  and  challenges  were  antic¬ 
ipated  before  the  effort  began,  many  were  dis¬ 
covered  through  false  starts,  dead  ends  and 
unsuccessful  runs.  Some,  no  doubt,  were 
missed  and  await  discovery  by  others.  Never¬ 
theless,  a  few  of  the  team's  observations  de¬ 
serve  consideration. 

Wargaming  simulations,  no  less  than  ana¬ 
lytical  models,  generate  enormous  quantities  of 
numerical  data.  It  is  tempting  to  manipulate 
and  interpret  wargaming  data  using  the  same 
techniques  so  useful  for  analytical  models. 
However,  quantitative  data  from  wargaming 
simulations  must  be  viewed  within  the  context 
of  the  test.14  Any  conclusions  derived  from  the 
data  must  acknowledge  the  characteristics  of 
wargaming-based  appraisals  when  compared 
to  more  typical  analytical  experiments.  Because 
the  central  focus  of  wargaming  is  the  human 
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Company-sized  force  of 
brigade  teams  encounters  an 
Acadian  battalion  moving  to  an 
objective. 

•  Brigade  mission  is  force  protection  and  then 
defeat  of  Acadian  forces. 

•  Acadian  mission  is  secure  movement  to  its 


Vignette  4 


Acadian  Maneuver 
Ads 


Brigade  Teams 


Figure  7.  Vignette  4. 


decision  processes,  each  instance  of  a  war  game 
is  unique.15  In  wargaming,  the  essential  as¬ 
sumption  for  sensitivity  analysis  that  demands 
everything  but  a  single  variable  be  held  con¬ 
stant  is  seldom  met.  For  example,  in  the  paired 
data  above,  the  paired  runs  were  alike  but  not 
identical.  Although  the  starting  conditions 
were  identical  except  for  the  value  of  the  fire 
support  variable,  once  the  runs  started  each 
team  had  a  different  set  of  decision  paths  avail¬ 
able.  Each  side  pursued  its  mission  based  upon 
the  assets  it  had  available  and  its  perception  of 
the  opponent's  reaction  to  its  decisions.  There¬ 
fore,  a  vignette  should  not  be  conceptualized  as 
a  linear  sequence  of  activities  through  time,  a 
single  thread  forming  a  decision  line.  Rather,  it 
is  a  multidimensional  decision  space  with  de¬ 
cision  nodes,  branches  and  sequels  forming  a 
response  surface.  Between  like  pairs  of  runs, 
some  of  the  branches  and  sequels  were  similar 
but  each  decision  space  was  unique.  Therefore, 
it  is  erroneous  to  assert  that  difference  in  run 


outcomes  (dependent  variables)  between  like 
pairs  is  the  sole  result  of  changes  in  the  inde¬ 
pendent  variables.  The  pairs,  while  similar,  are 
rigorously  speaking,  single  point  samples  from 
different  response  surfaces.  This  is  typical  of 
most  wargaming  but  leads  to  additional  com¬ 
plications  when  attempting  to  explain  out¬ 
comes. 

Many  models,  including  war  games,  em¬ 
ploy  stochastic  techniques  to  compute  such 
things  as  sensor  acquisition,  kills,  etc.  In  such 
cases,  an  observer  cannot  have  high  confidence 
that  a  single  sample  of  an  outcome  is  close  to 
the  expected  value.  To  achieve  a  higher  level  of 
confidence,  war  game  orders  are  saved  and  the 
model  is  run  numerous  times.  Theoretically, 
this  will  produce  a  series  of  outcomes  whose 
mean  approaches  the  expected  value.  This  tech¬ 
nique  is  not  valid  when  human  decision  pro¬ 
cesses  form  a  central  part  of  the  effort  because  it 
does  not  capture  the  branches  and  sequels  that 
might  have  been  generated  by  differing  values 
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Table  1.  Summary  of  Tests 
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1 

2 

3 

5  7 

9 

11 

13 

4 

6 

8 

10 

12 

14 

15 

16 

17 

18 

19 

20 

21 

22 

23 

24 

25 

26 
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1 

1 

2 

2  2 

2 

2 

2 

3 

3 

3 

3 

3 

3 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 
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X 
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X 

X  X 

X 
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X 
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X 

X 

X 

X 
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X 

X 

X 

X 

X 
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X 

X 

X 

X  X 
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X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Team  UAV 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Fires 

F/A-18 

X 

X 

X 

X  X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Arsenal  Ship 

X 

X 

X  X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Armed  UAV 

X 

X 

X 

X 

X 

X 

X 

X 

Sensor-shooter 

Time 

1  Min 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

11  Min 

X 

X 

X 

X 

X 

X 

X 

X 

"Winner" 

A 

A  US  A  A 

US 

? 

US  US  A 

A  US  US 

? 

US 

A 

A 

US  US 

A 

US 

A 

A 

US  US 

? 

Vignette  4  Data 

100 

90 
80 
70 
60 
50 
40 
30 
20 
10 
0 


□  Mission  Effectiveness  M  Unit  Survivability 


Normalized 
value  of 
Mission 
Effectiveness 
or  Unit 
Survivability 


15  16  17  18  19  20  21  22  23  24  25  26 

Run  Number 


Figure  8.  The  chart  shows  data  from  Vignette  4.  Mission  effectiveness  was  measured  as  the  amount  of 
adversary  personnel  or  systems  killed  or  destroyed,  normalized  between  0  (none)  to  100  (all).  Unit  survivability 
was  the  amount  of  brigade  team  personnel  and  systems  remaining,  normalized  between  0  (none)  to  100  (all). 


of  the  stochastic  event.  A  commander,  for  ex¬ 
ample,  may  alter  an  attack  (a  war  game  order) 
if  the  results  of  an  air  attack  (a  stochastic  event) 
were  exceptionally  effective.  One  compensation 
technique  is  to  examine  each  scenario  vignette's 
decision  space  and  identify  those  nodes  whose 
branches  are  primarily  determined  by  results 
from  stochastic  events.16  From  that  node,  then. 


additional  runs  can  be  conducted  to  see  if  sig¬ 
nificantly  different  branches  and  sequels  exist. 
In  the  JWFC  study  effort,  we  identified  one 
such  node,  the  effect  of  minefield  on  Acadian 
movement  during  the  ambush  in  Vignette  3.  In 
this  instance,  the  effectiveness  of  the  minefield 
changed  the  movement  rate  through  the  bri¬ 
gade  teams'  kill  zone.  It  did  not,  however,  cre- 


Military  Operations  Research ,  V3  N4  1997 


Page  33 


APPRAISING  WARFIGHTING  CONCEPTS  WITH  WARGAMING  SIMULATIONS 


Data  Organized  by  Variable 


Figure  9.  The  chart  shows  the  value  of  mission  effectiveness  from  vignette  pairs  in  which  all  the  independent 
variables  except  fire  support  were  treated  equivalently. 


Data  Organized  by  Variable 


Figure  10.  The  chart  is  similar  to  Figure  8  except  that  changes  in  unit  survivability  are  plotted  against  changes 
in  fire  support  for  equivalent  run  pairs. 

ate  significantly  new  decision  paths.  Exploring  ercise  design.  In  training  events,  wargaming 

significant  nodes  and  possible  decision  paths  is  scenarios  are  usually  broad  in  scope  and  time, 

laborious  but  it  can  be  minimized  through  ex-  If  wargaming  is  being  used  for  appraisals,  then 
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the  scenario  should  be  reduced  to  a  manageable 
number  of  small,  focused  scenario  vignettes 
each  containing  a  small  set  of  possible  decision 
paths. 

Compared  to  typical  uses  of  analytical 
models,  wargaming-based  concept  appraisals 
may  be  limited  in  depth  and  detail.  They  do, 
however,  exhibit  one  compelling  characteristic, 
the  inclusion  of  human  decision  processes.  To 
be  successful,  wargaming  appraisals  should: 

•  Be  used  when  human  decision  processes  are 
critical  components  of  the  study  issue — cre¬ 
ated  from  the  perspectives  of  a  commander 
and  staff  '. 

•  Have  a  small  number  of  variables — simplic¬ 
ity  provides  clarity. 

•  Be  focused — design  using  several  specific, 
limited  scenario  vignettes. 

•  Be  linked  to  appropriate  tactics,  techniques 
and  procedures — simply  grafting  new  weap¬ 
ons  into  existing  doctrine  is  not  appropriate. 

•  Present  synthesized  results — combine  expert 
observation  with  quantitative  measure¬ 
ments. 

Results  of  wargaming-based  appraisal  ef¬ 
fort  should  be  used  with  discretion.  Because 
they  lack  the  detail  of  analytical  models,  they 
are  not  appropriate  tools  for  proving  the  valid¬ 
ity  of  concepts  or  weapons  systems.  Rather, 
they  should  serve  as  method  of  obtaining  in¬ 
sight.  This  insight  can  then  point  to  areas  where 
more  detailed  analysis  or  assessment  may  be 
appropriate.  Some  examples  of  appropriate  use 
of  wargaming  appraisals: 

•  Exploring  technology-based  concepts  if  the 
primary  interest  is  the  linkage  or  interaction 
between  the  capability  of  systems  and  the 
pursuit  of  military  missions.  Wargaming 
can  investigate  how  new  employment  pat¬ 
terns  (and  adversary  reactions)  emerge  from 
human  decision  processes  that  respond  to 
changes  in  the  capabilities  of  weapons  sys¬ 
tems. 

•  A  filter  for  new  doctrine,  especially  those 
relying  on  information-based  systems  or  ca¬ 
pabilities.  Wargaming  can  be  used  to  antic¬ 
ipate  the  decision  space — nodes,  branches 
and  sequels — of  new  tactics,  techniques  and 
procedures  and  then  map  the  range  of  plau¬ 
sible  outcomes  under  various  scenarios. 

•  Establishing  parameters  for  operational 
field  tests  of  new  weapons  systems  or  doc¬ 
trine.  Wargaming  can  be  used  to  help  design 


the  scenario  elements,  range  space,  or  size  of 
an  impact  area  needed  to  test  a  new  weapons 
system  under  realistic  mission  conditions. 

Because  of  the  need  to  explore  multiple 
decision  paths,  wargaming  requires  a  large 
number  of  talented  military  professionals.  The 
JWFC  study  relied  on  fourteen  individuals  in 
the  design,  preparation,  execution  and  ap¬ 
praisal  stages  of  this  effort.  Four  critical  talents 
were  needed  and  all  team  members  were  ex¬ 
perts  in  at  least  two  of  them. 

•  Joint  warfighting  and  wargaming  experts — 
designed  and  executed  scenario  vignettes, 
created  brigade  team  composition,  weapons 
and  doctrine,  and  composed  the  command¬ 
ers'  mission  statements. 

•  Opposing  force  specialists — created  the  de¬ 
tails  for  the  experiment's  scenario,  a  logical 
opposing  force  campaign  plan,  and  individ¬ 
ual  supporting  missions. 

•  Operations  researchers — provided  the  over¬ 
all  experimental  design  and  synthesis  of  data 
and  observations. 

•  Technical  support  personnel — built  the  sim¬ 
ulation  database  and  configured  the  war 
game. 

Of  all  the  talents,  the  most  critical  was  joint 
warfighting  and  wargaming.  This  type  of  effort 
cannot  be  conducted  without  war  gamers  who 
are  proficient  in  current  and  emerging  joint  and 
service  doctrine,  tactics,  techniques  and  proce¬ 
dures.  In  addition,  the  war  gamers  must  be  well 
versed  in  the  characteristics  of  the  wargaming 
model.  Every  war  game  has  different  capabili¬ 
ties  and  limitations.  Professional  war  gamers 
can  compensate  for  the  inherent  inaccuracies 
and  limitations  of  the  simulation  without  sac¬ 
rificing  operational  military  validity. 

In  conclusion,  the  challenges  confronted  in 
using  a  wargaming  simulation  for  concept  ap¬ 
praisal  are  significant.  The  effort,  however,  can 
be  very  productive  because  it  is  a  methodology 
that  is  different  from  the  use  of  analytical  sim¬ 
ulations.  The  methodology  used  by  JWFC 
blended  techniques  from  training-based  war¬ 
gaming  events  with  those  used  for  analytical 
experiments.  Often  people  assume  there  is 
symmetry  between  simulation-assisted  exer¬ 
cises  and  analysis  if  they  both  use  computer 
models.  Such  an  assumption,  which  focuses  on 
the  tool  instead  of  the  task,  can  lead  to  flawed 
methodologies.  Understanding  the  difference, 
though,  can  provide  a  useful  tool  for  military 
operations  research. 
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ENDNOTES 

1  Throughout  summer  1996,  the  DSB  effort  en¬ 
gaged  numerous  participants  organized  into 
several  teams,  each  employing  a  variety  of 
methods  to  investigate  this  technologically 
based  warfighting  concept.  The  JWFC  study 
results  were  presented  to  DSB  Team  2,  Terri¬ 
tory  Control,  in  August  1996. 

2  Extracted  from  the  Memorandum  for  Chairman , 
Defense  Science  Board  [8].  To  frame  the  envi¬ 
ronment  and  characteristics  of  the  warfight¬ 
ing  concepts,  the  DSB  postulated  seven  dif¬ 
ferent  scenarios.  Because  of  time  and  resource 
constraints,  the  JWFC  effort  was  limited  to 
examining  one  scenario. 

3  The  JWFC  appraisal  team  was  composed  of  1 
military  and  14  contractor  personnel  from 
JWFC:  Maj  Greg  Brouillette,  Robert  Chap¬ 
man,  Steve  Corner,  Earl  Eaddy,  Doug  Failor, 
Robert  Fall,  Andre  Fryer,  Dave  Hastedt,  Nate 
Hilliard,  Ray  Lingo,  William  Pattison,  Larry 
Rautenberg,  Mark  Rose,  Larry  Stratton,  and 
John  Zanelli. 

4  Another  reason  for  limiting  the  reader's  ex¬ 
posure  to  the  study  is  to  preserve  the  confi¬ 
dence  of  the  original  customer.  This  is  the 
same  policy  JWFC  has  towards  its  exercise 
customers.  At  the  end  of  an  exercise,  a  Com¬ 
mander's  Summary  Report  is  provided  in 
confidence  to  the  exercise  sponsor.  It  is  not 
shared  with  superior,  lateral,  or  subordinate 
organizations.  Furthermore,  the  JWFC  team 
provided  appraisal  results  devoid  of  conclu¬ 
sions  or  recommendations.  It  was  left  to  the 
DSB  to  derive  insights  and  inferences  from 
the  data. 

5  The  basis  for  the  JWFC  concept  appraisal  pro¬ 
cess  was  based  on  the  Joint  Exercise  Life  Cy¬ 
cle  (JELC)  as  described  in  the  Joint  Training 
Manual,  [6,  Ch  VI].  The  JELC  is  a  disciplined 
process  to  design  an  exercise  that  links  mis¬ 
sion  essential  tasks  to  training  objectives  and 
ultimately  simulation  inputs  and  outputs. 
The  JWFC  study  team  created  the  depicted 
analogue  of  the  JELC  for  their  wargaming 


study.  Like  the  JELC,  the  appraisal  effort  is 
viewed  as  a  process  composed  of  discrete 
tasks  or  products  linked  together  and  orga¬ 
nized  into  several  progressive  stages. 

6  The  JWFC  concept  appraisal  is  different  from 
traditional  assessment  and  analyses.  While 
concept  appraisal  may  provide  insights,  it 
does  not  have  the  fidelity  of  analyses  or  the 
comprehensiveness  of  assessment.  Compared 
to  concept  appraisal,  assessments  are  more 
comprehensive  endeavors  whose  results  are 
often  used  for  evaluation  of  macro  issues 
such  as  force  structure  or  war  plans.  Analyses 
more  often  evaluate  micro  issues  through  de¬ 
tailed  reductions  of  complex  systems  into 
smaller  elements;  e.g.,  trade-off  studies  of  a 
weapons  system's  performance  characteris¬ 
tics. 

7  Although  more  restrictive  than  the  approved 
definitions,  they  are  consistent  with  DoD 
community  usage  where  the  terms  "model," 
"simulation"  and  "war  game"  are  often  used 
interchangeably.  In  reality,  they  have  differ¬ 
ent  meanings.  The  Glossary  of  Modeling  and 
Simulation  (M&S)  Terms  [5]  defines  war  game 
as  "a  simulation  game  in  which  participants 
seek  to  achieve  a  specified  military  objective 
given  pre-established  resources  and  con¬ 
straints;  for  example,  a  simulation  in  which 
participants  make  battlefield  decisions  and  a 
computer  determines  the  results  of  those  de¬ 
cisions."  Model:  "a  physical,  mathematical  or 
otherwise  logical  representation  of  a  system, 
entity,  phenomenon,  or  process."  Simulation: 
"a  method  for  implementing  a  model  over 
time." 

8  This  process  was  recorded  after  the  effort  and 
depicts  both  the  elements  the  appraisal  team 
accomplished  and  those  that  were  truncated 
because  of  the  lack  of  time.  Because  of  the 
time  constraint,  some  elements,  such  as  a 
Memorandum  of  Agreement,  were  not  at¬ 
tempted.  Others,  such  as  verification  and  val¬ 
idation,  were  combined. 

9  Verification  in  the  appraisal  process  focuses 
on  the  conceptual  model  of  the  warfighting 
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concept — the  database  and  simulation  at¬ 
tributes  and  their  ability  to  replicate  the  rele¬ 
vant  elements  of  the  concept.  This  includes 
the  creation  of  new  objects  and  modifying 
existing  objects  as  surrogates.  For  example,  in 
the  this  study  an  armed  UAV  was  created  by 
modifying  the  attributes  of  an  already  exist¬ 
ing  aircraft.  Overall,  this  activity  is  more  ver¬ 
ification  of  the  design  of  elements  than  code. 
Both  are  described  in  VV&A  Recommended 
Practices  Guide ,  [10,  Ch  1], 

10  Validation  follows  verification  but  is  more 
focused  on  results  from  practice  runs  of  the 
scenario.  Validating  data  and  model  func¬ 
tionality  for  concepts,  tactics,  and  technolo¬ 
gies  that  have  yet  to  be  fielded  operationally 
(or  even  tested)  is  an  effort  to  check  for  rea¬ 
sonable  behaviors  and  results.  It  is  not  an 
effort  to  define  the  accuracy  of  the  results. 
The  best  one  can  hope  for  is  selective  in¬ 
sights  into  a  few  relationships,  not  predictive 
results,  marginal  values,  or  tradeoffs.  Again, 
refer  to  VV&A  Recommended  Practices  Guide . 

11  The  warfighting  concept  explored  by  the 
DSB  Summer  Study  was  similar  to  ideas 
present  in  Joint  Vision  2010,  the  Chairman's 
". . .  operationally-based  template  for  the 
evolution  of  the  Armed  Forces  for  a  chal¬ 
lenging  and  uncertain  future"  [7].  Like  the 
DSB  Summer  Study,  Joint  Vision  2010  ex¬ 
plores  future  concepts  for  military  opera¬ 
tions.  There  are,  however,  significant  differ¬ 
ences.  While  Joint  Vision  2010  includes  all  the 
Services,  the  DSB  Summer  Study  focused  on 
land  forces.  Both  examine  relationship  be¬ 
tween  information  superiority,  technological 
innovations  and  operational  concepts.  But 
again.  Joint  Vision  2010  has  a  wider,  opera¬ 
tional  focus  whereas  the  DSB  study  was 
technologically  oriented  to  more  specific  ap¬ 
plications. 

12  Refer  to  The  Universal  Joint  Task  List  (UJTL) 
for  an  explanation  of:  the  strategic,  opera¬ 
tional,  and  tactical  levels  of  war;  the  process 
used  to  analyze  a  mission  into  a  series  of 
related  tasks;  and  identifying  mission  essen¬ 
tial  tasks  and  assigning  appropriate  condi¬ 


tions  and  standards.  Understanding  the 
UJTL  approach  is  essential  to  wargaming. 

13  The  challenges  faced  by  the  JWFC  study 
team  were  not  unique.  Battillega  and  Grange 
in  The  Military  Applications  of  Modeling  de¬ 
scribe  these  difficulties.  "The  challenge  and 
art  of  modeling  is  the  representation  of  pri¬ 
mary  relationships,  where  the  question  of 
relevance  is  entirely  dependent  on  the  pur¬ 
pose  for  which  the  model  is  used."  [1,  page 
7]  And  also:  "The  model  must  ultimately 
resolve  the  conflict  between  the  desired  level 
of  modeling  detail  and  uncertainties  which 
exist  in  basic  input  data.  Sometimes  the  res¬ 
olution  transfers  the  burden  to  the  analysts 
in  the  sense  that  they  can  either  extrapolate 
details  into  basic  data  so  that  they  can  gen¬ 
erate  appropriate  model  inputs,  or  attempt 
to  interpolate  between  model  results  focused 
too  imprecisely  to  produce  desired  analysis 
results."  [1,  page  9] 

14  Although  presented  in  a  slightly  different 
context,  Robert  Elberth  makes  a  cogent  ar¬ 
gument  that,  with  respect  to  detail,  fidelity, 
realism  and  accuracy,  ". . .  reality  doesn't  ex¬ 
ist  within  the  computer."  [4,  page  26]  Many 
simulations,  such  as  the  one  used  for  this 
appraisal,  JCM,  can  portray  very  detailed 
combat  activities  (e.g.,  the  field  of  fire  of  an 
infantry  rifleman  in  an  urban  environment 
based  on  the  interaction  between  weapons 
range  and  line-of-sight  limitations  due  to  ob¬ 
struction).  Detail,  however,  does  not  imply 
validity.  "There  are  real  risks — potentially 
measured  in  lives  lost — to  pushing  detail 
and  causal  relationships  below  the  data  for 
which  we  have  real  data." 

15  In  a  training  event,  each  instance  of  an  event 
is  unique — it  can  be  done  only  once.  One  of 
the  most  powerful  tools  in  quantitative  anal¬ 
yses  is  the  ability  to  run  an  experiment  nu¬ 
merous  times  to  derive  expected  outcomes, 
establish  the  range  of  possible  outcomes, 
and  modify  variables  for  sensitivity  analysis. 
The  technique  is  not  possible  when  real  hu¬ 
man  decision  processes  are  used.  The  limi¬ 
tation  is  both  practical  and  theoretical.  Sim- 
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ulation  events  using  real  commanders  and 
staffs  are  expensive  and  time  consuming. 
Most  joint  force  commanders  cannot  assem¬ 
ble  their  staffs  and  components  together  for 
a  training  event  more  than  once  a  year.  Even 
then,  the  event  must  be  carefully  tailored 
and  reduced  in  scope  to  focus  on  the  most 
pressing  objectives.  Even  if  it  were  possible 
to  engage  these  staffs  more  often,  theoretical 
limitations  apply.  Commanders  and  staffs 
learn.  Playing  the  same  scenario  many  times 
with  the  same  group  is  not  the  same  as  run¬ 
ning  a  simulation  many  times.  In  a  strict 
sense,  human  decision  processes  are  uncon¬ 
trollable  variables.  Whenever  they  are  part 
of  the  experiment,  each  instance  of  the 
''same"  experiment  is  really  a  single  sample 
from  different  universes.  This  characteristic 
invalidates  the  application  of  most  statisti¬ 
cally  based  sensitivity  analyses. 

16  The  concept  of  a  commander's  decision 
space  is  critical  to  wargaming.  For  a  concise 
description  of  how  decision  spaces,  courses 
of  action  and  planning  horizons  relate  to 
war  gaming  see  Time  and  Command  Opera¬ 
tions:  The  Strategic  Role  of  the  Unified  Com¬ 
mands  and  the  Implications  for  Training  and 
Simulations.  [11,  pages  31-35]  The  authors  of 
this  IDA  report  were  focused  at  the  unified 
and  sub-unified  command  level.  However, 
the  same  attributes,  appropriately  scaled, 
apply  to  tactical-level  decision  processes.  Al¬ 
though  the  JWFC  study  was  conducted  be¬ 
fore  their  document  was  written,  it  articu¬ 
lates  many  of  the  experimental  design 
concepts  applied  by  the  JWFC  appraisal 
team. 
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ABSTRACT 

his  paper  reports  on  the  use  of  virtual 
simulation  for  concept  evaluation.  It 
focuses  on  excursions  conducted  at 
the  Institute  for  Defense  Analyses  (IDA) 
Simulation  Center  for  the  1996  Defense  Sci¬ 
ence  Board  (DSB)  Summer  Study  on  Tactics 
and  Technology  for  21st  Century  Military 
Superiority.  These  excursions  engaged  a 
slice  of  a  2015  battlefield  portraying  only 
the  targeting  elements  of  two  small  teams 
(2-3  men  each)  plus  an  intermediate  head¬ 
quarters  and  a  task  force  headquarters.  The 
exercise  was  divided  into  excursions  de¬ 
signed  to  investigate  combat  effectiveness 
resulting  from  parametric  variations  in 
small  team  size  and  composition,  mission, 
organic  sensor  capabilities,  and  remote  sen¬ 
sor  suites.  The  teams  interfaced  with  a  syn¬ 
thetic  battlefield,  visually  through  3D  visu¬ 
alization  portals  and  electronically  through 
sensor  and  communications  interfaces.  Fu¬ 
ture  doctrine  was  initially  developed  and 
incrementally  changed  from  lessons 
learned  during  the  excursions.  The  project 
produced  results  on  the  evaluation  of  bat¬ 
tlefield  concepts  and  emerging  technology 
and  on  the  evaluation  of  advanced  distrib¬ 
uted  simulation's  utility  for  analysis. 


KEYWORDS 

Advanced  distributed  simulation,  vir¬ 
tual,  concept  evaluation,  synthetic  bat- 
tlespace 

INTRODUCTION 

The  evaluation  of  concepts  is  challeng¬ 
ing  to  the  analysis  community  because  of 
the  difficulty  in  optimizing  the  effects  of 
future  technology,  doctrine,  organization, 
or  training.  Evaluation  of  concepts  involv¬ 
ing  small  units  on  the  battlefield  is  partic¬ 
ularly  difficult.  Traditionally,  analysis  has 
been  performed  with  constructive  models 
and  simulations.  However,  analysts  know 
that  behavioral  algorithms  in  constructive 
models  are  suspect  for  examining  future 
concepts,  especially  those  involving  phe¬ 
nomena  not  validated  in  the  models.  Such 
phenomena  include  information  domi¬ 
nance  or  military  operations  against  new 
threats.  Virtual  simulation  may  provide  a 
new  approach  to  concept  evaluation.  Inter¬ 
esting  aspects  of  this  approach  include  the 


use  of  "human-in-the-lo op "  ( virtual )  simu¬ 
lation  and  a  synthetic  battlespace  (distrib¬ 
uted,  entity-based,  and  real  time  within  a 
realistic  environment).  It  was  tested  as  a 
proof  of  principle  in  the  analysis  of  con¬ 
cepts  and  technologies  investigated  by  the 
1996  DSB  Summer  Study.  A  description  of 
the  DSB  analysis  needs  and  the  scenario 
developed  for  responding  to  these  needs,  a 
description  of  the  virtual  simulation  design 
and  excursions,  and  finally  a  presentation 
of  analysis  of  the  observations  with  some 
results  about  virtual  simulation  and  the 
synthetic  battlespace  in  support  of  concept 
evaluation  is  provided. 


CHALLENGES  FOR  CONCEPTS 
MODELING 

Army  acquisition  policy  (DA  Pam  70-3) 
describes  concepts  as  key  components  of 
top  down  cycle  guidance.  Warfighting  con¬ 
cepts  based  on  future  threats  and  technolog¬ 
ical  forecasts  provide  a  means  for  identifying 
and  analyzing  future  required  capabilities. 

A  cursory  review  of  the  literature  on 
military  modeling  (e.g.  Battilega  &  Grange, 
Bracken  et.  al.,  and  Przemieniecki)  reveals 
surprisingly  little  discussion  on  the  model¬ 
ing  of  concepts.  This  could  be  indicative  of 
a  belief  among  analysts  that  there  is  no 
difference  in  modeling  concepts  vice  mod¬ 
eling  the  current  state.  The  author's  expe¬ 
rience  indicates  that  current  modeling  prac¬ 
tices  assume  that  future  concepts  are 
incremental  variants  of  current  warfighting 
capabilities  and  therefore  can  be  modeled 
by  incrementally  varying  current  models. 

The  concepts  explored  by  the  1996  DSB 
Summer  Study  on  Tactics  and  Technology 
for  21st  Century  Military  Superiority  and  dis¬ 
cussed  in  this  paper  were  not  incremental 
variants  of  the  current  state.  Rather  they  were 
"quantum  leaps"  from  the  present  into  the 
future.  How  does  one  analyze  such  future 
concepts? 

The  analysis  of  concepts  requires  that 
we  think  about  the  limits  of  conceptualiza¬ 
tion.  [Sowa]  discusses  the  incongruity  of 
the  continuous  aspects  of  the  real  world 
with  the  discrete  nature  of  conceptual 
structures  for  artificial  intelligence.  This  is 
one  challenge  to  analyzing  future  warfight¬ 
ing  concepts.  For  example,  the  DSB  con¬ 
cepts  were  discontinuous  with  current  doc¬ 
trine,  technologies,  and  organizations.  The 
use  of  existing  continuous  aggregated  con¬ 
structive  simulations  may  add  no  value  to 
understanding  these  concepts  since  these 
simulations  have  no  validity  in  replicating 
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future  worlds.  Indeed,  the  number  of  simula¬ 
tions  available  to  address  information  technol¬ 
ogy  issues  in  current  operational  context  is  lim¬ 
ited.  Many  constructive  simulations  are  simply 
incompatible  with  the  operational  impacts  of 
new  information  based  technologies  and  can¬ 
not  be  configured  rapidly  to  accommodate  new 
doctrines  and  organizations. 


A  CONCEPT  OF  SMALL  UNITS  ON  A 
FUTURE  BATTLEFIELD 

The  DSB  focused  . .  on  the  concept  of 
making  relatively  small  and  rapidly  deployable 
forces  (or  teams) — specially  equipped,  trained, 
and  supported  by  remote  sensors  and  weap¬ 
ons — able  to  accomplish  missions  heretofore 
only  possible  with  much  larger  and  massed 
forces." 

The  DSB  considered  concepts  where  battle¬ 
field  tasks  are  redefined  so  that  the  infantry¬ 
man's  mission  of  "closing  with  and  killing  the 
enemy"  was  changed  to  "managing  remote 
sensors  in  the  detection,  classification,  identifi¬ 
cation  of  targets,  and  surviving."  These  con¬ 
cepts  required  higher  level  command  to  "tag 
and  track"  targets  for  optimal  engagement  by 
precisely  guided,  remotely-based  indirect  fire. 
They  also  included  the  idea  of  empowerment — 
giving  the  soldier  the  information  he  required 
whenever  he  asked  for  it,  and  consequently 
giving  him  increased  authority  in  his  actions. 

Technologies  occupying  a  central  position 
in  the  DSB  study  included  assured  stealthy 
communications  such  as  digital  cellular;  a 
global  information  infrastructure  including  a 
common  grid  reference  (geolocation,  terrain, 
features),  connectivity  (reliable,  low  probability 
of  intercept,  voice,  data,  &  video),  and  database 
fusion  and  analysis;  massive  numbers  of  re¬ 
mote  sensors  such  as  micro-UAVs  and  unat¬ 
tended  ground  sensors;  massive  availability  of 
remotely  and  precisely  delivered  indirect  fires 
such  as  arsenal  ship  or  intelligent  minefields; 
stealthy  vehicles  such  as  all  electrical  for  sur- 
vivable  mobility;  and  "tagging  and  tracking" 
technologies  such  as  micro-electromechanical 
systems  (MEMs)  or  micro-UAVs.  The  DSB 
Summer  Study  also  explored  additional  con¬ 
cepts  and  technologies  not  discussed  here.  Fu¬ 
ture  warfighting  requirements  will  depend  on 
regional  situations  that  vary  greatly  on  mis¬ 
sions,  geo-political  considerations,  and  geogra- 
phy. 


The  changing  role  of  warfare  analysis  is 
that  while  monolithic  Lanchester  based  models 
might  be  defended  for  analyzing  cold  war  sce¬ 
narios  where  a  Soviet  threat  was  poised  to 
thrust  massive  numbers  of  tanks  through 
NATO  forces,  they  cannot  be  defended  for 
analysis  in  the  emerging  international  security 
environment.  Geographically,  defense  concerns 
are  now  more  globally  diffused.  Regional  con¬ 
flict  implies  increased  variability  in  threat  ca¬ 
pabilities  and  includes  the  idea  of  an  asymmet¬ 
rical  threat  where  low  technology  capabilities 
are  leveraged  against  high  technology  vulnera¬ 
bilities.  Missions  assigned  to  theater  forces  are 
of  a  broader  spectrum  including  operations 
other  than  war.  Also,  technological  capabilities 
are  more  information-based,  enabling  im¬ 
proved  warfighting  capabilities  such  as  mass¬ 
ing  fires  vice  massing  forces.  All  of  these  factors 
indicate  the  need  for  improved  capabilities  for 
analysis. 

Representative  analyses  were  conducted  by 
different  organizations  for  the  DSB  including 
the  following  topics. 

•  The  impact  of  employing  small  teams  and 
managing  fires. 

•  The  required  time  line  for  indirect  fires. 

•  Base  case  and  effect  of  expanding  situational 
awareness  and  creating  tracks  between 
teams. 

•  Effect  of  direct  control  of  UAV  sensors  by 
team  and  insight  on  ground-based  coopera¬ 
tive  engagement  capability. 

•  Impact  of  mobility  and  sensor  capabilities  on 
team  performance. 

•  Mobility  impact  on  effectiveness  of  teams, 
plus  helicopter  survivability. 

•  Effect  of  better  situational  awareness  of 
weapon  mix,  kills,  and  losses. 

This  article  is  concerned  with  the  first  of 
these  analyses  which  was  unique  in  its  focus  on 
the  soldier  and  on  its  application  of  simulation 
type — virtual  simulation. 

EXPERIMENTAL  DESIGN 

[Ackoff]  describes  the  following  roadmap 
for  the  research  process: 

1.  Formulating  the  problem 

2.  Constructing  the  model 

3.  Testing  the  model 

4.  Deriving  a  solution  from  the  model 

5.  Testing  and  controlling  the  solution 
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6.  Implementing  the  solution 

Each  step  of  this  roadmap  is  addressed  in  the 
sections  below.  However,  there  are  important 
aspects  of  the  experimental  design  that  require 
elaboration  and  qualification. 

The  first  aspect  is  that  this  was  a  "proof-of- 
principle"  effort.  The  experiment  was  designed 
to  accommodate  many  different  variables  such 
as  different  numbers  of  sensors  and  weapons 
with  varying  performance  characteristics,  dif¬ 
ferent  organizations,  different  doctrines  and 
tactics,  different  terrain  environments,  and  dif¬ 
ferent  team  equipment  sets.  We  did  not  attempt 
a  full  factorial  experiment  in  an  effort  to  isolate 
variables  to  which  combat  results  might  be 
most  sensitive  nor  did  we  fully  control  the  so¬ 
lutions  from  the  experiment.  For  example,  only 
eleven  excursions  were  executed. 

Notwithstanding,  we  followed  an  ap¬ 
proach  that  has  been  promoted  by  [Landauer 
and  Nielsen]  called  User-Centered  Design  or 
Usability  Engineering  in  which  the  usability  of 
a  system  was  emphasized  over  its  technical 
specifications.  In  doing  so,  we  relied  on  the 
powerful  cognitive  capabilities  of  our  "usabil¬ 
ity  testers"  for  resolving  the  complexities  that 
result  from  having  to  deal  with  so  many  vari¬ 
ables  and  so  many  unknowns  that  accompany 
the  exploration  of  concepts.  In  this  approach, 
Nielsen  describes  a  moderately  different  model 
in  the  design  of  an  experiment. 

Pick  existing  user  interfaces.  Subject  them  to  a 
simple  user  test  by  defining  some  typical  test  tasks 
and  observe  users  as  they  try  performing  the  tasks 
with  the  system.  Where  problems  are  found  resolve 
them  through  iterative  design. 

The  second  aspect  is  the  validation  of  the 
simulation  design.  It  was  believed  that  the  ex¬ 
periment  required  a  scenario  that  would  pro¬ 
vide  different  features  (environmental,  time- 
distance,  logical  consistency  in  events, . . .)  and 
would  support  a  list  of  functions  (battlefield 
surveillance,  requests  for  fire,  management  of 
sensors, . . .).  The  simulation  needed  to  support 
a  scenario  that  stimulated  future  warfighting 
processes.  The  experiment  used  terrain  and  en¬ 
tities  developed  for  Synthetic  Theater  of  War- 
Europe  (STOW-E)  in  October  1994.  Although 
we  had  access  to  the  more  robust  terrain  data 
bases  (dynamic  objects,  weather,  smoke,  etc) 
being  developed  for  STOW  97,  we  decided  to 
use  STOW-E  databases  instead  because  of  their 


proven  stability.  Subject  matter  experts  in  the 
experiment  team  determined  that  the  STOW-E 
databases  were  credible  for  the  applications  en¬ 
visioned  for  the  virtual  experiment.  As  with 
most  validation  efforts,  a  decision  was  reached 
on  the  degree  of  validation  that  could  be  af¬ 
forded  vis  a  vis  the  benefits  expected  from  the 
experiment.  This  decision  included  scoping  the 
experiment  so  that  some  simulation  capabilities 
such  as  the  resolution  of  the  synthetic  environ¬ 
ment  would  not  be  required  for  the  success  of 
the  experiment.  It  was  known  but  accepted  that 
the  coarse  resolution  of  terrain  would  impact 
conclusions  about  small  team  survivability  (e.g. 
impact  on  line  of  sight).  Rather  than  opt  for 
higher  resolution  synthetic  terrain,  we  chose  to 
exclude  survivability  of  small  teams  as  an  issue 
for  our  scenario  design  (i.e.  the  synthetic  terrain 
stability  decision  above).  In  addition  to  the  val¬ 
idation  of  synthetic  forces  and  environments 
used  in  the  experimental  design,  validation  of 
performance  characteristics  of  weapons  and 
sensor  systems  was  conducted  through  liaison 
with  authoritative  experts  from  OSD  or  the  Ser¬ 
vices.  Concepts  from  the  Marine  Corps  Sea 
Dragon  and  the  Army's  Army  After  Next  Pro¬ 
gram  were  used  to  generate  initial  battlefield 
processes  for  examination.  Finally,  the  simula¬ 
tion  design  was  reviewed  by  selected  members 
of  the  DSB  panel  periodically  during  its  devel¬ 
opment  to  assure  that  the  experiment  would 
demonstrate  what  was  expected  of  it. 

The  third  aspect  is  the  question  of  why  the 
experiment  could  not  have  been  run  in  an  ag¬ 
gregate  level  constructive  simulation.  In  the  past, 
aggregate  level  constructive  simulations  have 
been  used  because  of  their  ability  to  run  faster 
than  real  time,  allowing  frequent  variable  value 
changes  for  sensitivity  analysis.  However,  Davis 
and  Blumenthal  provide  illustrations  where  "ag¬ 
gregated  models  are  invalid  [with  respect  to]  phe¬ 
nomena  omitted  or  buried  [such  as]  command- 
control  processes,  tactics,  . . .  and  other  "soft 
factors' . . . ."  Further,  Dreyfus  raises  questions 
with  respect  to  whether  computers  will  ever  be 
able  to  process  information  of  certain  cognitive 
behaviors,  pointing  to  a  growing  body  of  evi¬ 
dence  that  human  and  mechanical  information 
processing  proceed  in  entirely  different  ways  (e.g. 
learning,  vision,  natural  language  understanding, 
problem  solving,  and  pattern  recognition).  In 
other  words,  human  behavior  is  difficult  to 
model.  General  Larry  Welch,  retired  Air  Force 
Chief  of  Staff  has  stated  on  occasion,  "Construc¬ 
tive  simulations  model  the  past  while  the  past  is 
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not  a  good  predictor  of  the  future/'  noting  further 
that  the  range  of  warfighting  capabilities  and  sce¬ 
narios  is  becoming  increasingly  wide  for  both  the 
United  States,  its  allies,  and  its  potential  foes. 
Nonetheless,  the  "entity-based"  (vice  aggregate 
level)  constructive  ModSAF  proved  crucial  to  the 
success  of  this  virtual  experiment.  ModSAF  en¬ 
abled  the  interoperability  of  humans,  their  mis¬ 
sion  equipment,  and  the  experimental  compo¬ 
nents,  including  the  simulation.  It  also  provided 
the  mechanism  for  incorporating  future  equip¬ 
ment  design. 

Finally  cost  of  a  virtual  simulation  is  an 
issue.  The  project  was  given  a  specific  amount 
of  money,  much  of  which  was  used  for  invest¬ 
ment  in  equipment  that  would  later  be  used  for 
other  analyses  and  much  of  which  was  paid  for 
DSB  activities  that  were  independent  of  the 
virtual  experiment.  It  is  difficult  to  put  an  exact 
dollar  figure  on  the  cost  of  the  experiment. 
Within  the  current  state  of  the  art,  systems  en¬ 
gineering  is  a  costly  aspect  of  initiating  virtual 
simulations.  However,  the  author  believes  this 
aspect  of  cost  should  be  considered  as  transi¬ 
tory.  Dramatic  improvements  in  processing 
and  communications  and  the  evolution  of  stan¬ 
dards  in  advanced  distributed  simulation 
promises  to  reduce  costs  over  the  near  future. 
For  example,  STOW  97  used  $5,000  Pentium 
workstations  to  drive  ModSAF  and  synthetic 
environments,  where  $35,000  SGI  workstations 
were  required  for  STOW-E.  If  Moore's  Law 
continues  to  prevail  through  next  year,  $2,500 
workstations  may  do  the  same  job  twice  as 
well.  Moreover,  initial  testing  of  STOW  has 
already  demonstrated  that  organic  C4I  systems 
(AFATDS,  MCS-P,  GCCS  or  JMCIS,  and 
CTAPS)  can  be  integrated  effectively  with  a 
simulation  system.  Further,  it  is  clear  that  op¬ 
portunities  exist  for  STOW  in  the  improvement 
of  inefficiencies.  The  STOW  technologies  that 
include  the  use  of  artificial  intelligence,  natural 
language  interfaces,  and  other  labor  saving 
techniques  should  facilitate  the  reduction  of 
manpower  in  future  simulation  design.  In  other 
words,  the  state  of  the  art  is  moving  rapidly 
towards  a  simulation  infrastructure  that  will 
support  the  development  of  experiments  re¬ 
quiring  less  systems  engineering.  On  the  other 
hand,  analysis  using  virtual  simulation  should 
be  expected  to  be  persistently  more  costly  in 
manpower  per  unit  run  time  period  than  anal¬ 
ysis  using  constructive  simulations.  In  this  ex¬ 
periment,  we  used  seven  individual  combat¬ 
ants,  four  observers  (two  operational  and  two 


behavioral),  and  six  controllers.  Other  experi¬ 
ments  will  require  more  or  less  depending  on 
the  nature  of  the  problems  under  investigation. 
When  making  such  comparisons,  it  is  astute  to 
recognize  that  there  are  some  important  issues 
that  best  can  be  analyzed  through  virtual  sim¬ 
ulation,  particularly  usability  or  interface  prob¬ 
lems.  In  comparison,  a  field  experiment  should 
be  expected  to  cost  much  more  than  a  virtual 
experiment  in  examining  some  issues  involving 
human  performance  and  provide  much  less 
flexibility.  To  confirm  such  beliefs  [Worley,  et 
al.]  reported  that  the  United  States  Atlantic 
Command's  AGILE  PROVIDER  94  Exercise 
cost  $48  million,  while  the  United  Endeavor  95 
exercise,  using  the  Joint  Training  Confederation 
of  constructive  simulations  cost  only  $3.4  mil¬ 
lion  or  about  7%  of  the  cost  of  AGILE  PRO¬ 
VIDER  involving  the  same  training  audience. 
While  this  comparison  was  made  in  a  training 
environment  and  certainly  was  not  an  exhaus¬ 
tive  study,  it  is  illuminating.  In  order  to  inves¬ 
tigate  issues  from  this  experiment  in  a  field 
exercise,  important  components  of  a  JTF  and  an 
associated  OPFOR  would  need  to  be  fielded. 
Even  then,  fielding  of  new  technologies  would  be 
exorbitant  in  cost  or  impossible  to  field  because 
they  were  still  in  development.  On  balance,  we 
expect  that  a  field  exercise  would  be  more  suit¬ 
able  than  a  virtual  experiment  in  examining  is¬ 
sues  involving  physical  and  mental  stress. 

[Ackoff]  points  out  that  research  is  directed 
toward  the  solution  of  problems  of  two  major 
classes:  evaluative  (also  called  summative  evalua¬ 
tion  by  the  instructional  community)  and  devel¬ 
opmental  (also  called  formative  evaluation).  Sum¬ 
mative  evaluation  determines  how  good  a 
system  is  after  it  has  been  developed,  but  not 
particularly  useful  in  producing  information 
about  how  to  make  a  system  in  development 
more  useful.  Formative  evaluation  provides  in¬ 
formation  on  what  needs  fixing,  amplification, 
or  replacement.  The  DSB  virtual  simulation  in¬ 
volved  both  types  of  evaluation  but  clearly  was 
more  focused  on  the  second  type  of  problem. 


SCENARIO  DESIGN 

The  objective  of  the  virtual  simulation 
project  was  to  demonstrate  some  aspect  of  con¬ 
cept  exploration  that  could  only  be  analyzed 
through  virtual  simulation,  i.e.  could  not  be 
better  analyzed  through  constructive  or  live 
simulation.  With  this  end  in  mind,  a  scenario 
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was  designed  which  was  consistent  with  com¬ 
bat  scenarios  under  consideration  by  the  DSB. 
This  scenario,  illustrated  in  Figure  1,  postulated 
early  entry  into  some  littoral  area  of  the  world. 
Small  teams  would  be  inserted  by  helicopters 
from  a  marine  based  platform,  supported  by 
precision  indirect  fire  from  an  arsenal  ship  or 
other  remote  locations  including  Navy  based 
air,  attack  helicopters,  or  MLRS  across  the  in¬ 
ternational  border.  The  mission  of  the  team  was 
to  interdict  the  movement  of  enemy  forces 
through  a  designated  terrain  area  to  support 
sealing  off  an  objective  area  being  secured  by 
other  forces  (e.g.  an  airfield).  It  was  assumed 
for  the  purposes  of  this  analysis  that  local  se¬ 
curity  of  the  team  would  be  provided  by  indig¬ 
enous  forces.  Survivability  of  the  forces,  al¬ 
though  an  important  issue  to  the  new  concepts, 
was  a  secondary  element  for  this  analysis.  The 
team  was  provided  a  vehicle  to  move  on  the 


battlefield  although  some  excursions  examined 
dismounted  operations.  The  team  had  no  or¬ 
ganic  capability  to  engage  targets  with  direct 
fire.  The  only  attack  capability  available  was 
indirect  fire.  The  team  was  provided  commu¬ 
nications  with  a  common  data  base  that  pro¬ 
vided  geo-location  information  on  a  digital  ter¬ 
rain  backdrop  (map  of  the  future  (MOF))  and 
had  a  digital  messaging  system  (GRUNT)  for 
reporting  information  and  requesting  fires. 
Various  sensor  arrays  were  used  during  the 
excursions  that  included  Melios  (binoculars  in¬ 
tegrated  with  a  laser  range  finder),  COVER  (a 
locally  tethered  sensor  that  hovered  about  200 
feet  above  the  team's  position  like  a  battlefield 
periscope  for  looking  over  hills),  and  access  to 
sensor  data  that  would  be  controlled  either  by 
the  Joint  Task  Force  Commander  or  higher  au¬ 
thorities.  Some  of  the  general  issues  under  in¬ 
vestigation  included  "Why  do  we  need  infan- 


Figure  1.  The  Basic  Elements  of  the  Scenario  Analyzed. 
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trymen  on  the  ground  at  all?"  "Given  this 
need — what  will  empowerment  of  the  infantry¬ 
man  do  for  the  operation? — Will  it  indeed  make 
the  operation  more  effective  or  will  the  soldier 
on  the  ground  be  overpowered  with  informa¬ 
tion  management  demands?"  The  weapons  and 
their  characteristics  postulated  for  this  analysis 
are  shown  in  Table  1.  Weapons  were  selected  to 
represent  what  would  be  reasonably  available 
for  the  early  entry  scenario  being  designed. 
LTG  Paul  Van  Riper,  USMC  cited  to  the  DSB  a 
need  to  reduce  the  time  interval  between  call  for 
fire  and  weapons  impact  to  less  than  2-5  minutes 
as  an  objective  for  small  team  survivability.  A 
loitering  missile  which  was  called  Super  T-Hawk 
was  postulated  to  satisfy  this  need. 

Ordnance  for  the  Arsenal  Ship  was  postu¬ 
lated  as  the  Navy  Tactical  Missile  System 
(NTACMS)  having  roughly  the  same  characteris¬ 
tics  as  the  ATACMS.  It  is  believed  that  the  un¬ 
classified  performance  characteristics  shown  on 
the  table  for  all  of  the  systems  are  achievable 
realistically  within  the  current  state  of  technol¬ 
ogy. 

The  sensor  suite  used  within  the  designed 
scenario  and  the  characteristics  of  each  sensor 
are  shown  in  Table  2.  In  order  to  achieve  the 
level  of  situational  awareness  required  by  the 
concepts  being  considered  by  the  DSB,  a  set  of 


notional  sensors,  as  shown  in  this  table  with 
their  characteristics,  were  postulated.  As  with 
the  notional  weapons,  the  notional  sensors 
were  given  capabilities  that  could  reasonably 
be  expected  with  current  or  emerging  technol¬ 
ogy.  Further,  it  was  postulated  that  a  common 
database  such  as  tactical  internet  would  be 
available  for  accessing  sensor  data  from  theater 
or  national  sensors. 


SIMULATION  DESIGN 

The  physical  layout  of  the  simulation  facil¬ 
ity  is  shown  in  Figure  2.  All  of  the  components 
(exercise  control,  observers,  and  simulation 
synthetic  battlespace  portals)  were  in  one  large 
room,  while  exercise  control  stations  were  sep¬ 
arated  physically  from  portals  and  observers. 
The  term  "portal"  was  used  by  the  DSB  to  mean 
the  interface  between  an  individual  combatant 
(IC)  and  the  synthetic  environment.  This  usage 
may  cause  some  confusion  with  the  virtual  sim¬ 
ulation  community,  to  be  clarified  below. 

There  were  three  portals.  Portal  1  was  a 
three  walled  "synthetic  environment  cave" 
around  a  treadmill  that  allowed  a  walking  or 
crawling  soldier  to  interact  directly  with  the 
synthetic  battlespace.  Some  use  the  term  "im- 


Figure  2.  Simulation  Facility  Layout. 
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Table  1.  Weapons  Postulated  for  the  Simulation 


Characteristic 

Weapons  Types 

Units 

NTACM 

Super 

T-Hawk 

TACAIR 

F-16 

How 

155mm 

MLRS 

ATK 

AH64 

Naval 

Gun 

Rounds  per  volley 

Rds 

na 

na 

na 

8 

8 

na 

4 

Aircraft  per  flight 

A/C 

na 

na 

2 

na 

na 

4 

na 

Speed 

kph 

1500 

500 

2000 

1000 

1000 

180 

1000 

Maximum  Range 

km 

150 

500 

1000 

30 

30 

500 

30 

Launch  interval 

min 

2 

30 

0 

1 

1 

0 

1 

On  station  time 

min 

na 

60 

20 

na 

na 

45 

na 

Number  of  return 
passes 

# 

0 

0 

3 

na 

na 

4 

na 

Interval  between  passes 
Communications 

Intervals 

min 

na 

na 

5 

na 

na 

2 

na 

FDC  to  launcher 

min 

10 

5 

5 

2 

2 

5 

8 

Launcher  to  launch 

min 

3 

0 

0 

2 

2 

0 

2 

'  Point  Target  Warhead 

MP 

MP 

MP 

Chead 

MP 

Hellfire 

DPICM 

Implemented  in 
ModSAF  by 

bomb 

bomb 

bomb 

155  imp. 

MLRS 

Hellfire 

bomb 

0.5 

Laser  designated  Pk 
for  tracks 

Pk 

0.2 

0.2 

0.8 

0.5 

0.7 

0.8 

Laser  designated  Pk 
for  wheels 

Pk 

0.4 

0.4 

0.9 

0.2 

0.7 

1 

0.5 

Laser  designated  Pk 
for  troops 

Pk 

0.4 

0.4 

na 

na 

na 

na 

na 

Area  Target  Warhead 

MP 

MP 

MP 

DPICM 

MP 

Hydra  70 

HE 

Implemented  in 
ModSAF  by 

bomb 

bomb 

bomb 

155  prox 

MLRS 

bomb 

bomb 

Footprint  against 
tracks 

m-x-m 

400X400 

200X200 

200X200 

200X200 

200X200 

100X100 

400X400 

Pk  against  tracks  in 
footprint 

Pk 

0.2 

0.2 

0.2 

0.1 

0.1 

0.1 

0.1 

Footprint  against 
wheels 

m-x-m 

400X400 

200X200 

200X200 

200X200 

200X200 

100X100 

400X400 

Pk  against  wheels 
in  footprint 

Pk 

0.4 

0.4 

0.3 

0.2 

0.2 

0.4 

0.2 

Footprint  against 
troops 

m-x-m 

400X400 

200X200 

200X200 

200X200 

200X200 

100X100 

400X400 

Pk  against  troops  in 
footprint 

FASCAM 

Mines  installed  by 
ModSAF 

Pk 

0.5 

0.5 

0.6 

0.4 

0.4 

0.8 

0.4 

Footprint 

400X400 

na 

400X400 

100X100 

100X100 

na 

na 

Density  (mines  per 
footprint*) 

Target  Location  Update 

m-x-m 

400 

na 

500 

6.5 

100 

na 

na 

Update  window 
(before  impact) 

sec 

120-60 

120-60 

300-240 

na 

na 

60-30 

na 

Correction  envelope 

km 

3 

na 

10 

na 

na 

2 

na 

Laser  designate 
(impact — X) 
Unattended  Ground 
Sensor 

sec 

na 

7 

7 

7 

7 

na 

na 

Num.  per  sortie 

# 

0 

0 

0 

0 

0 

4 

0 

Launch  interval  of  "0"  indicates  continuous  availability.  HE  =  High  Explosive.  Chead  =  Copperhead.  Pk  = 
probability  of  kill  given  an  attack/per  pass  or  vehicle.  FASCAM  =  Field  Artillery  Scatterable  Mines.  DPICM  = 
Dual  Purpose  Improved  Conventional  Munitions.  MP  =  Multipurpose. 
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Table  2.  Sensor  Suite 


Sensor 

Type 

Range 

Coverage 

Target  Data 

IFF 

Platform 

Downlink 

Existing  Sensors  (Operational  &  Developmental) 

REMBASS 

Magnetic 

3/25  m 

Continuous 

MTI  digital 

no 

manpack 

Monitor  Set 

REMBASS 

Acoustic 

50/350  m 

Continuous 

MTI  digital 

no 

manpack 

Monitor  Set 

REMBASS 

IR  Passive 

3/50  m 

Continuous 

digital 

no 

manpack 

Monitor  Set 

JSTARS 

SAR 

240  km 

11  hrs 

MTI /FTI  imagery 

no 

E-8A 

GSM* 

Predator 

Radar 

500  nm 

24  hrs 

MTI  digital 

no 

UAV 

GSM* 

Predator 

E-O/IR 

500  nm 

24  hrs 

MTI/ FTI  dig. /video 

limited 

UAV 

GSM* 

Tier  II  + 

Radar/E-O/IR  3000  nm 

24  hrs 

MTI/FTI  multi  media 

limited 

UAV 

GSM* 

Tier  III  - 

SAR  or  E-O 

500  nm 

8  hrs 

MTI  /FTI  multi  media 

limited 

UAV 

GSM* 

Notional  Sensors 

TF  UAV 

FLIR 

100  km 

Continuous 

MTI/FTI  digital 

limited 

UAV 

MOF 

SAR 

digital 

limited 

LLTV 

video 

limited 

REMBASS 

Magnetic 

£ 

O 

o 

X 

o 

o 

Continuous  MTI /FTI 

UGS 

MOF 

II 

Acoustic 

IR 

LLTV 

Micro 

10  km 

4  hrs 

MTI/FTI  video 

some 

UAV 

MOF 

UAV 

FLIR 

digital 

some 

COVER 

Radar 

Continuous  MTI /FTI  digital 

Tethered 

GSM 

LLTV 

video 

Hovercraft 

FLIR 

digital 

REMBASS  =  Remotely  Monitored  Battlefield  Sensor.  LLTV  =  Low  level  televideo.  EO  =  Electro-optical.  MTI  = 
Moving  Target  Indicator.  FTI  =  Fixed  Target  Indicator.  UAV  =  Unmanned  aerial  vehicle.  GSM  =  Ground 
Station  module.  SAR  =  Synthetic  aperture  radar.  FLIR  =  forward  looking  infrared.  IR  =  Infrared. 

*  Initially  will  be  High  Altitude  Endurance  (HAE)  common  ground  segment  for  both  Tier  II  and  III. 


mersion"  to  described  this  situation  where  an 
IC  is  surrounded  by  a  synthetically  generated 
environment.  Portal  2,  illustrated  in  Figure  3, 
was  the  driving  portal,  where  the  IC  interacted 
with  "windows"  into  the  environment,  but  did 
not  have  a  complete  presence  within  the  envi¬ 
ronment.  This  is  the  situation  which  the  com¬ 
munity  uses  the  term  "portal"  in  contrast  to 
"immersion."  Initially,  this  portal  was  manned 
with  one  driver  and  one  forward  observer  who 
observed  the  battlefield  and  initiated  calls  for 
fire.  Workload  analysis  revealed  that  the  task 
load  was  too  great  for  two  people,  so  a  third 
one  was  added  to  share  in  the  tasks.  Specifi¬ 
cally,  it  took  too  long  for  the  forward  observer 
both  to  observe  the  battlefield  and  process  calls 
for  fire.  The  illustration  shows  what  equipment 
interfaced  with  the  synthetic  battlespace.  These 
included  controls  for  a  wheeled  vehicle  such  as 
the  High  Mobility  Multi-purpose  Wheeled  Ve¬ 
hicle  (HMMHV),  a  foot  pedal  for  dismounted 
operations,  a  MOF  that  provided  a  surrogate 
for  future  digital  maps  (1:50,000,  scaleable  to 


1:100,000  and  1:250,000),  and  a  digital  message 
pad  for  electronic  message  forms.  The  digital 
MOF  showed  the  location  of  all  other  known 
friendly  and  enemy  forces,  enabling  the  evalu¬ 
ation  of  situational  awareness.  Also  provided 
were  a  radio  for  communications  with  higher 
headquarters  and  a  surrogate  sensor  interface 
for  locally  controlled  sensors  such  as  Melios 
(laser  range  finder  integrated  with  binoculars). 
Portal  3  was  the  intermediate  leader  portal  that 
included  a  desk  top  monitor  of  the  synthetic 
battlespace  integrated  to  a  MOF  display.  It 
showed  team  and  enemy  locations  and  permit¬ 
ted  communications  via  electronic  messaging. 
A  detailed  view  of  how  the  IC  interfaced  with 
the  virtual  battlefield  is  illustrated  in  Figure  4. 

Exercise  control  was  facilitated  through 
three  control  stations,  one  of  which  is  illus¬ 
trated  in  Figure  5.  The  Remote  Fire  Station  in¬ 
cluded  the  interfaces  shown.  A  spreadsheet 
based  fire  control  emulator  was  used  to  elec¬ 
tronically  track  data  on  weapons  used  for  en¬ 
gagement.  Each  fire  event  was  recorded,  in- 
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Figure  4.  Interfacing  the  Soldier  with  the  Virtual  Battlefield. 
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eluding  the  weapon  identification,  the 
remaining  inventory,  and  the  time  line  for  the 
engagement.  The  "entity-based"  ModSAF  was 
used  as  the  simulation  engine  for  the  exercise. 
The  ModSAF  bomb  button  was  used  to  execute 


indirect  fire  missions.  A  MOF  or  GRUNT  was 
used  to  receive  calls  for  fire.  (Both  had  this 
capability).  The  MOF  was  used  to  keep  track  of 
the  location  of  friendly  and  enemy  forces  on  the 
battlefield.  The  Remote  Fire  Station  was 
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Figure  5.  Remote  Fire  Station. 


manned  by  two  Marine  Corps  Captains  from 
the  USMC  Artillery  School  and  another  indi¬ 
vidual  managed  the  fire  control  emulator. 

The  other  two  control  stations  were  config¬ 
ured  similarly,  without  a  fire  control  emulator. 
The  Remote  Sensor  Station  operator  used  a 
ModSAF  terminal  to  replicate  and  control  sen¬ 
sor  activity  within  the  simulation.  Required 
sensor  capabilities  that  exceeded  the  current 
inventory  of  ModSAF  sensor  models  were  sim¬ 
ulated  off-line  and  the  results  of  their  returns 
were  manually  entered  into  the  common  data¬ 
base  accessed  by  the  MOF.  A  MOF  terminal 
also  was  located  at  the  Remote  Sensor  Station 
for  monitoring  friendly  and  enemy  locations. 

The  Exercise  Director  used  a  ModSAF  ter¬ 
minal  at  his  station  to  monitor  enemy  and 
friendly  locations  against  the  desired  scenario 
for  each  excursion.  The  Exercise  Director  also 
served  as  the  Commander  of  Higher  Headquar¬ 
ters  in  receiving  and  responding  to  requests 
from  the  small  teams. 

The  illustration  in  Figure  6  shows  the  orga¬ 
nizational  relationship  between  individual 
combatant  (IC),  observer,  controller  personnel, 
and  their  respective  stations — physically  shown 
in  Figure  2. 


EXCURSIONS 

After  two  months  of  systems  design  and 
engineering,  the  simulation  exercise  was  exe¬ 
cuted  over  a  period  of  two  weeks  in  July  1996. 
Marine  Corps  lieutenants  acted  as  ICs  in  this 


simulation  for  one  of  these  weeks,  and  Army 
captains  manned  the  simulation  as  ICs  in  the 
other  week.  Monday  of  each  of  the  two  weeks 
was  devoted  to  training  the  ICs  on  simulation 
use  and  postulating  new  doctrine  and  tactics. 
Friday  of  these  weeks  was  dedicated  to  after 
action  review  (AAR).  Two  AAR  sessions  were 
held  daily.  One  included  exercise  controllers 
and  ICs.  The  other  included  only  controllers. 
Two  excursions  were  run  on  the  other  three 
days;  one  in  the  morning  and  another  in  the 
afternoon.  A  detailed  schedule  of  configuration 
variables  by  excursion  is  shown  at  Table  3.  An 
equipment  outage  precluded  one  scheduled  ex¬ 
cursion.  The  table  is  self-explanatory,  however, 
we  can  highlight  some  of  the  issues  that  drove 
variable  change. 

Two  terrain  types  were  considered — open 
and  mixed.  Urban  terrain  was  also  considered 
and  demonstrated,  but  resource  constraints 
limited  activities  in  the  urban  setting.  Another 
constraining  factor  was  the  additional  equip¬ 
ment  that  would  have  been  required  for  urban 
operations.  The  mixed  terrain  provided  more 
adequate  and  interesting  situations  than  open 
terrain  for  small  teams  because  of  the  greater 
complexity  that  it  presented  for  information 
management.  Sensor  and  communications  ca¬ 
pabilities  were  varied.  On  one  excursion, 
ground  truth  information  was  provided  to  the 
small  team  to  investigate  the  upper  limit  of 
information  access.  These  variables  impacted 
situational  awareness  greatly. 

The  mobility  and  missions  of  enemy  forces 
were  varied  somewhat  but  not  enough  to  make 
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Figure  6.  The  Functional  Design  of  the  Exercise. 


significant  observations  about  the  use  of  small 
teams  versus  differing  enemy  capabilities. 

A  weapons  array  was  used  that  varied  re¬ 
sponsiveness,  accuracy,  area  of  impact,  lethal¬ 
ity,  and  control. 


DATA  COLLECTION  AND  ANALYSIS 

An  evaluation  plan  was  developed  around 
three  essential  elements  of  analysis  (EE A). 

1.  The  utility  of  sensors  and  personal  data  as¬ 
sistants  (GRUNT  and  MOF)  in  enhancing 
small  team's  situational  awareness. 

2.  The  utility  of  remote  fires  in  increasing  the 
combat  effectiveness  of  small  teams. 

3.  The  suitability  of  virtual  simulation  as  a  con¬ 
cept  exploration  tool. 

These  EEA  were  used  for  defining  measures  of 
effectiveness  and  performance  for  analysis. 


Sensors  and  Personal  Data 
Assistants 

The  utility  of  sensors  was  measured 
through  two  sets  of  data.  The  first  set  involved 
the  amount  of  data  made  available  to  the  small 
teams  because  of  the  availability  of  increasing 
sensing  capability.  These  data  related  to  im¬ 
pacts  (workload  and  performance)  caused  by 
increasing  amounts  of  information.  The  second 
set  involved  the  requirement  to  manage  new 
pieces  of  equipment  (sensors,  MOF,  and  PDAs). 
These  data  related  to  their  usability  and  their 
impact  on  roles,  responsibilities,  procedures. 


Remote  Fires 

The  utility  of  fires  was  measured  by  small 
team  effectiveness.  Measurements  included  num¬ 
bers  of  enemy  killed  and  time  to  kill.  The  num¬ 
bers  killed  measured  general  system  effective- 
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Table  3.  Simulation  Excursions 


Excursion 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

Environment 

Rural 

Rural 

Rural 

Rural 

Rural 

Rural 

Rural 

Rural 

Rural 

Rural 

Rural 

Terrain 

M 

M 

M 

M 

M 

O 

O 

M 

M 

M 

M 

Portal  I  (Treadport) 

MOF 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

GRUNT 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Melios 

X 

X 

X 

X 

X 

X 

Magnification 

1-3 

1-3 

1-6 

1-6 

COVER 

X 

X 

X 

X 

X 

Magnification 

1-6 

1-6 

1-6 

1—6 

1-6 

Elevation  Rate 

50  m/s 

50  m/s 

50  m/s 

50  m/s 

100  m/s 

100  m/ 

Mobility- 

D 

D 

D 

D 

D 

V 

D 

V 

V 

V 

V 

Portal  2  (Driving  Portal) 

MOF 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

GRUNT 

X 

X 

X 

X 

X 

X 

X 

X 

Melios 

X 

Magnification 

1-6 

COVER 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Magnification 

1-3 

1-3 

1-3 

1-6 

1-6 

1-6 

1-6 

1-6 

1-6 

1-6 

Elevation  Rate 

50  m/s 

50  m/s 

50  m/s 

50  m/s 

50  m/s 

50  m/s 

50  m/s 

50  m/s 

100  m/s 

100  m/ 

Mobility 

V 

V 

V 

V 

V 

V 

D 

V 

D 

V 

V 

Team  Leader  Portal 

None 

None 

None 

MOF 

X 

X 

X 

X 

X 

X 

X 

X 

Mobility 

V 

V 

V 

D 

V 

D 

V 

V 

UGS  (Hand 

2 

2 

2 

2 

2 

2 

2 

4 

4 

4 

4 

Emplaced) 

Remote  Sensors 

Task  Force  UAV 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

JSTARS 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Tier  11/111 

X 

X 

X 

X 

X 

X 

X 

X 

Predator 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Ground  Truth 

X 

Weapons 

NTACMS 

X 

X 

X 

X 

X 

X 

X 

X 

Super  T-Hawk 

X 

X 

X 

X 

X 

X 

X 

X 

TAC  AIR 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Howitzer 

X 

MLRS 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

ATK  HELO 

X 

X 

X 

X 

Naval  Gunfire 

X 

X 

X 

Enemy  Forces 

Mission 

I 

I 

I 

I 

I 

R 

R 

Vehicle  Speed  (kph) 

15 

15 

15 

15 

15 

15 

15 

15 

15 

29 

29 

Tank  Probes 

9 

2 

2 

4 

4 

6 

6 

4 

4 

4 

5 

BMP  Probes 

11 

3 

3 

5 

5 

6 

6 

4 

4 

5 

6 

Truck  Probes 

2 

1 

1 

2 

2 

Dismt'd  Probes 

19 

4 

4 

8 

8 

4 

4 

5 

5 

8 

11 

I  =  Infiltration.  MOF  =  Map  of  the  Future.  GRUNT  =  Electronic  message  pad.  D  =  Dismounted.  UGS  = 
Unattended  Ground  Sensor.  Melios  =  Binoculars  integrated  with  rangefinder.  V  =  Vehicle.  NTACMS  = 
Arsenal  Ship  Tactical  Msl.  COVER  =  Sensor  tethered  to  individual  combatant.  R  =  Reconnaissance.  M  =  Mixed 
Terrain.  O  -  Open  Terrain.  Super  T-Hawk  =  Loitering  Tomahawk. 


ness.  Time  to  kill  measured  system  efficiency  compared  with  aggregated  constructive  or  live 
particularly  with  respect  to  team  survivability.  simulation  for  modeling  the  DSB  concepts. 

As  the  experiment  was  designed,  con¬ 
structed,  tested,  and  executed  four  natural  ar- 

Suitability  of  the  Virtual  Simulation  eas  for  making  such  judgment  emerged. 

The  data  required  to  determine  suitability 

consisted  of  that  data  necessary  to  make  judg-  1.  Concept  exploration — What  capabilities 

ments  about  the  benefits  of  virtual  simulation  does  virtual  simulation  uniquely  provide? 
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2.  Human  performance — What  human  perfor¬ 
mance  does  virtual  simulation  measure? 

3.  Process  Analysis — What  insight  does  virtual 
simulation  provide  about  battlefield  pro¬ 
cesses  that  could  be  automated? 

4.  System  Specification — How  can  virtual  sim¬ 
ulation  be  used  best  to  transform  require¬ 
ments  into  system  design? 

Data  required  to  answer  the  concept  explo¬ 
ration  EEA  and  these  questions  on  suitability 
were  collected  in  the  categories  shown  in  Table 
4  using  the  methods  and  analyzing  the  func¬ 
tions  shown.  The  Army  Research  Institute 
(ARI)  developed  the  data  collection  and  analy¬ 
sis  plan  for  human  behavior.  ARI  also  partici¬ 
pated  in  the  exercise  as  expert  observers  in 
human  behavior.  This  plan  was  based  on  data 
requirements  for  analyzing  the  interfaces  be¬ 
tween  humans  and  equipment  such  as  the 
MOF,  workload,  and  human  performance  ARI 
also  helped  assess  doctrinal,  tactical,  and  pro¬ 
cedural  functions. 

[Salter,  Knerr,  et  al.]  discuss  in  detail  the 
ARI  participation  in  this  experiment.  The  ARI 


report  includes  its  data  collection  plan  includ¬ 
ing  measures  and  their  use  in  responding  to  the 
EEA. 

One  important  human  performance  issue 
that  emerged  was  the  interface  between  the  ICs 
and  their  equipment.  In  illustration,  it  was 
found  that  a  critical  requirement  exists  to  more 
efficiently  enter  and  extract  information  from 
automated  devices  such  as  voice  recognition  or 
touch  screens  in  order  to  make  these  devices 
credibly  functional.  Since  the  software  engineer 
who  had  originally  designed  some  of  the  equip¬ 
ment  interfaces  to  the  IC  was  available  during 
the  experiment,  we  were  able  to  react  immedi¬ 
ately  to  recommended  interface  improvements 
from  the  IC,  effectively  improving  interfaces  in 
near  real  time. 

The  entity-based  simulation  approach  im¬ 
proved  control  of  data,  leveraging  the  data  log¬ 
ging  capabilities  of  ModSAF.  Data  logging  al¬ 
lows  the  recall  of  any  event  (platforms,  activity, 
time,  location)  recorded  by  the  simulation. 
Other  automatic  data  collection  was  performed 
by  recording  data  managed  by  other  equip- 


Table  4.  Data  Collection 


Category  &  EEA 

Collection  Methods 

Functions 

Behavior 

Direct  Observation 

Human 

EEA-CE1,  CE2,  SI,  S2, 

Questionnaire 

•  Interface 

S3,  S4  (Notes) 

Interviews 

•  Workload 

•  Performance 

Doctrine 

Observation 

Roles  &  Responsibilities 

EEA-1,  SI 

Questionnaire 

Organization 

Interviews 

Operations 

Equipment 

The  Methods  above 

Data  presentation  &  use 

EEA-1  &  2 

plus 

Sensor  performance 

Automatic  data 
collection 

Database  management 

Procedures 

The  Methods  above 

Comm  procedures 

EEA  1  &  2,  SI 

plus 

Managing — Targets  &  Tasks 

Voice  log 

Battle  Damage 

Assessment 

Tactics 

All  the  Methods 

FASCAM  Fires 

EEA  1  &  2 

above 

Mobility 

Sectors 

Situational  Awareness 

Training 

All  the  Methods 

Target  acquisition,  tracking,  &  engagement 

EEA  1  &  2,  S2 

above 

Communications 

Managing  sensors 

Calls  for  fire 

Situational  Awareness 

Notes:  CE  =  Concept  Exploration  EEA.  S  =  Virtual  Simulation  Suitability  EEA. 
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ment  used  in  the  exercise.  For  instance,  the  fire 
control  emulator  maintained  a  comprehensive 
record  of  each  fire  engagement  including  the 
weapons  used  and  their  timeline. 

Remote  Fires 

In  retrospect,  the  indirect  fire  support  time¬ 
lines  probably  were  the  most  globally  useful 
data  set  collected  of  any  concept  investigated. 
(See  the  snapshot  of  data  in  Figure  7  below). 

Timeline  compression  was  critical  to  the 
survivability  of  small  teams  under  the  scenarios 
postulated  and  to  effective  Joint  Task  Force 
(JTF)  engagement  of  targets.  Elaborating  on  the 
survivability  issue,  indirect  fires  were  the  only 


means  of  protection  for  small  teams  when  they 
became  detected  by  an  armored  threat.  The 
need  to  reduce  the  'Time-of-flight"  to  2-5  min¬ 
utes  became  an  objective  for  small  team  surviv¬ 
ability.  However,  none  of  the  postulated  indi¬ 
rect  systems  were  able  to  deliver  ordnance 
within  5  minutes  of  target  detection.  It  appears 
that  this  problem  might  be  overcome  by  in¬ 
creasing  the  number  of  a  loitering  missiles, 
thereby  reducing  the  probabilistic  range  of  the 
missile  and  therefore  "time-of-flight."  Improve¬ 
ment  of  the  inflight  update  of  missiles  already 
on  the  way  might  also  be  a  solution  to  the 
" time-of-flight"  challenge.  It  would  help  in  the 
engagement  of  targets  that  had  moved  since  the 
call  for  fire  was  initiated  and  would  allow  ICs 


i  •  m I 

i  i  rr  i  i  r 

-15  -10  -5  0  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15 
Note:  Missing  markers  indicate  that  the  event  was  not  observed. 


MLRS  Trial  11  -  Event  1 
!  Super  T  Trial  10  -  Event  1 

I  MLRS  Trial  10  -  Event  2 
TACAIR  Trial  10  -  Event  3 
TACAIR  Trial  10  -  Event  4 


•  Appear  on  the  Map  of  the  Future  (MOF)  =  Target  Cued  by  Remote  Sensor 

•  Within  Line  of  Sight  (LOS)  =  First  instance  of  LOS  between  target  and  observer 

•  Request  permission  to  fire  =  Team  Leader  cannot  determine  if  a  target  is  within  rules 

of  engagement  and  request  approval  from  higher  headquarters. 

•  Receive  permission  to  fire  =  Higher  headquarters  approves  request 

•  End  of  Call  for  Fire  (CFF)  =  Last  key  stroke  or  voice  command  of  the  CFF. 

•  On  the  way  -  FDC  acknowledgement,  with  impact  time  and  munitions  type. 

•  Target  loaction  update  =  Team  reports  updated  location  of  target  prior  to  missile  impact. 

(Required  for  NTACMS  and  TAC  AIR  only.) 

•  Munitions  impact  =  munitions  land  on  target 

•  Battle  Damage  Assessment  (BDA)  =  Team  reports  target  damage. 


Figure  7.  Indirect  Fire  Timelines. 
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to  engage  new  targets  that  had  become  a 
greater  threat  to  the  ICs'  survivability  after  the 
initial  call  for  fire. 

The  snapshot  data  shown  in  Figure  7  re¬ 
veals  another  important  point.  The  team  was 
unable  to  render  effective  battle  damage  assess¬ 
ment  (BDA)  on  most  enemy  targets  attacked. 
Targets  were  usually  engaged  out  of  the  team's 
field  of  view,  i.e.  by  the  time  of  weapon's  im¬ 
pact,  they  had  moved  out  of  the  field  of  view. 


CONCEPT  EVALUATION  RESULTS 

In  general,  it  was  found  that  there  was  a 
limit  to  the  amount  of  information  that  an  in¬ 
dividual  could  manage.  Options  to  this  situa¬ 
tion  required  reducing  areas  of  responsibility, 
relying  on  greater  teamwork,  and  reapportion¬ 
ing  tasks  between  the  teams  and  the  Joint  Task 
Force.  It  was  observed  that  the  design  of  equip¬ 
ment  used  in  the  experiment  was  optimized  for 
individual,  not  team  use.  For  example,  report 
formats  in  equipment  menus  required  duplica¬ 
tion  of  data  entry  by  different  individuals  in  the 
chain  of  people  requesting  fire.  Also,  some  of 
the  equipment  designed  for  situational  aware¬ 
ness  (e.g.  personal  data  assistant  map  viewers) 
did  not  display  a  sufficiently  large  field  of  view 
to  optimize  their  situational  awareness.  While  it 
was  earlier  envisioned  that  dismounted  teams 
could  be  effective  in  some  missions,  it  was 
found  that  a  team  needed  a  vehicle  in  all  of  the 
situations  exercised  to  be  effective.  In  the  sce¬ 
narios  postulated,  the  battlefield  operational 
tempo  was  too  great  for  dismounted  teams  to 
keep  up  with  the  pace  of  enemy  movement  and 
survive. 

Four  specific  areas  for  results  included  sen¬ 
sor  management,  weapons  management,  data 
management,  and  data  presentation.  They  were 
organized  in  these  broad  categories  to  be  con¬ 
sistent  with  the  other  analyses  being  conducted 
for  the  DSB. 

•  Sensor  Management — Key  team  functions 
should  be  to  detect,  classify,  and  identify 
enemy  forces  not  observable  by  other  sensor 
systems  and  to  determine  enemy  intent. 
Even  where  enemy  forces  are  observable  by 
other  sensor  systems,  human  intervention 
may  be  required  to  manage  sensor  position 
for  observing  critical  aspects  of  targets.  Cog¬ 
nitive  capabilities  of  humans,  "locally  at  the 
scene"  can  be  indispensable  in  sensor  man¬ 
agement. 


•  Weapons  Management — Teams  had  diffi¬ 
culty  managing  more  than  two  targets  at  the 
same  time.  Survivability  of  the  small  teams 
require  the  tagging,  tracking,  and  engaging 
of  targets  within  2-5  minutes  of  detection. 
Finally,  returning  to  previous  target  for  up¬ 
date  may  cause  the  loss  of  other  targeting 
opportunities.  (A  corollary  is  that  small 
teams  should  not  be  managing  fires,  rather 
should  limit  their  activities  to  locally  manag¬ 
ing  sensors  for  the  tagging  of  targets  and  to 
selected  BDA  tasks.) 

•  Data  Management — Technology  should  en¬ 
hance  C2  by  providing  teams  the  right  infor¬ 
mation  when  needed.  The  use  of  common 
data  bases  that  can  be  accessed  by  multiple 
teams  or  individual  proved  quite  efficient  for 
information  exchange. 

•  Data  Presentation — Digital,  scaleable  maps 
of  appropriate  size  that  can  perform  distrib¬ 
uted  automated  battle  management  and  ter¬ 
rain  analysis  are  required  for  control  of  large 
areas.  Conversely,  it  was  found  that  digital 
maps  having  reduced  coverage  were  of  lim¬ 
ited  use  to  small  team.  It  was  important  for 
the  small  team  to  be  aware  of  the  situation  as 
seen  a  couple  of  levels  above,  not  merely  the 
weapons  range  of  the  small  team. 


FINDINGS  ABOUT  THE  USE  OF 
VIRTUAL  SIMULATION  FOR 
CONCEPT  EVALUATION 

Findings  on  the  utility  of  virtual  simulation 
are  organized  into  four  categories — concept  ex¬ 
ploration,  human  performance,  process  analy¬ 
sis,  and  system  specification.  All  of  the  findings 
are  presented  in  terms  of  benefits  provided  by 
virtual  and  entity-based  simulations  in  compar¬ 
ison  with  aggregated  constructive  simulations. 

•  Concept  Exploration — Virtual  simulation 
provides  an  effective  environment  for  creat¬ 
ing  and  experimenting  with  new  doctrine, 
tactics,  and  techniques.  New  capabilities  in¬ 
clude  the  synthetic  battlespace,  providing 
variability  of  environments;  and  entity-based 
simulation,  providing  a  mechanism  for  cre¬ 
ating  future  combat  systems,  interoperability 
between  models,  and  modularity  in  simula¬ 
tion  and  analysis  design. 

•  Human  Performance — Virtual  simulation 
enables  the  investigation  of  individual  work¬ 
load  (e.g.  task  and  information  management. 
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skill  definition;  including  teamwork  and 
cognitive  impacts),  roles  (e.g.  sensor, 
shooter,  fighter  tradeoff),  and  equipment  in¬ 
terface. 

•  Process  Analysis — Virtual  simulation  is  an 
appropriate  technique  for  assessing  new  bat¬ 
tlefield  processes  such  as  sensor  and  weap¬ 
ons  management.  Sound  process  analysis  as¬ 
sures  that  we  apply  new  technology  to  new 
processes  when  appropriate,  rather  than  au¬ 
tomate  obsolete  processes. 

•  System  Specification — Development  of  the 
virtual  simulation  design  requires  under¬ 
standing  of  included  processes  such  as  fire 
support  to  the  detail  necessary  to  accurately 
specifying  requirements  for  system  design — 
the  crucial  detail  being  the  human  as  an  el¬ 
ement  of  the  combat  system. 

If  these  findings  are  comprehensive  in 
some  way,  then  the  utility  of  virtual  simulation 
for  concept  exploration  could  be  limited  to  the 
investigation  of  information  management  is¬ 
sues.  While  this  exercise  did  not  fully  investi¬ 
gate  these  limits,  it  seems  likely  that  the  exam¬ 
ination  of  physics  based  phenomena  may  better 
be  pursued  using  other  techniques  such  as  con¬ 
structive  simulation. 


SUMMARY 

It's  not  clear  from  this  experiment  that  we 
fully  answered  general  issues  such  as  "Why  we 
must  have  infantrymen  on  the  ground  in  the 
future?",  although  we  were  able  to  provide  in¬ 
sight  and  surface  evidence  about  information 
processing  capabilities  yielded  from  individu¬ 
als'  cognitive  abilities  that  cannot  realistically 
be  expected  from  technology  over  the  next 
twenty  years.  In  order  to  resolve  better  such 
issues  further  experimentation  is  required. 

On  the  other  hand,  the  experiment  did 
demonstrate  great  potential  for  using  virtual 
simulation  in  concept  exploration  especially  for 
assessing  human  interfaces  to  complex  infor¬ 
mation  systems.  One  major  benefit  in  virtual 
simulation  over  aggregated  constructive  and 
live  simulation  included  the  ability  to  quickly 
and  economically  create  variation  in  a  future 
environment  (threats,  visibility,  geography, 
technologies,  organization,  doctrine,  and  tac¬ 
tics).  Another  benefit  was  in  enabling  investi¬ 
gation  of  what  arguably  is  the  most  important 
and  complex  part  of  any  battlefield  system — 
the  human  element.  While  live  simulation  also 


can  be  used  for  this  purpose,  live  simulation 
generally  is  more  costly  and  it  cannot  provide 
the  variability  required  for  future  concepts  nor 
can  it  integrate  the  variables  effectively  in  the 
controlled  way  enabled  by  virtual  simulation. 

There  also  appears  to  be  a  special  benefit  in 
using  advanced  distributed  simulation  in  sim¬ 
ulation  for  experimental  design.  Entity-based 
simulation  facilitates  interoperability  of  mod¬ 
els,  supports  comprehensive  data  collection 
and  analysis,  and  enables  the  articulation  of 
futuristic  systems  that  can  be  designed  at  an 
appropriate  level  of  fidelity. 

This  experiment  did  not  fully  test  the  limi¬ 
tations  of  virtual  simulation  for  concept  explo¬ 
ration.  While  this  experiment  demonstrated 
value  for  exploring  complexity,  the  author  be¬ 
lieves  that  analysis  of  simpler  problems  such  as 
those  involving  the  physics  of  weapons  sys¬ 
tems  might  best  be  pursued  with  closed  form 
analytic  models  which  are  repeatable,  deter¬ 
ministic,  and  can  be  supported  by  stable  theory. 

Future  work  should  examine  the  relation¬ 
ships  between  virtual,  live,  and  constructive 
simulation  for  analysis.  More  work  is  sug¬ 
gested  for  the  design  of  experiments  that  sup¬ 
port  usability  engineering,  particularly  using 
virtual  simulation.  The  development  of  tools  to 
improve  usability  engineering  also  seems  to  be 
a  worthwhile  objective. 
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ABSTRACT 

his  paper  introduces  a  new  aggregate 
measure  of  cost  and  operational  ef¬ 
fectiveness,  the  Cost  Exchange  Ratio. 
Aggregate  measures  have  been  used  for 
many  years,  but  have  been  primarily  con¬ 
cerned  with  operational  effectiveness.  We 
develop  the  basic  formalism,  which  is  ame¬ 
nable  to  any  combat  modeling  or  simula¬ 
tion  effort  analysis.  Next,  we  present  an 
example  using  Lanchester  attrition  theory 
to  demonstrate  the  utility  of  the  method  in 
concepts  analysis  and  evaluation.  Finally, 
we  establish  a  baseline  Cost  Effectiveness 
Ratio  using  Formal  Aggregation  Theory  as 
a  methodology  aggregation. 


I.  INTRODUCTION 

Recently,  military  services'  downsizing 
has  given  rise  both  to  accelerated  acquisi¬ 
tion  mechanisms  such  as  Advanced  Tech¬ 
nology  Demonstrations  (ATDs)  and  Rapid 
Acquisition  Programs  [Army  S&T  Master 
Plan,  1995]  and  to  greater  emphasis  on  user 
involvement  and  life  cycle  modeling,  sim¬ 
ulation,  and  analysis  (MSA).  Punctuated 
MSA  efforts  such  as  Trade-Off  Analyses/ 
Determinations  and  Cost  and  Operational 
Effectiveness  Analyses  (COEAs)  have  ei¬ 
ther  fallen  into  disuse  or  suffered  massive 
changes  of  character  as  the  role  of  joint 
requirements  intensifies. 

These  changes,  and  the  increased  num¬ 
ber  of  technological  concepts  have  in¬ 
creased  the  need  for  MSA  to  support  deci¬ 
sion  making  at  all  levels  from  the  local 
Program  Element/Project  Manager 
through  senior  service  leadership.  The 
needs  of  these  decision  makers  for  metrics 
seems  to  enjoy  a  degree  of  commonality 
albeit  for  somewhat  different  reasons.  At 
the  local  level,  the  limitations  of  resources 
and  the  large  number  of  concepts  dictates  a 
need  for  timely  and  simple  but  accurate 
estimations  of  the  concepts'  military  value 
and  cost.  At  the  senior  level,  the  same  con¬ 
siderations,  intensified  by  sheer  workload 
and  evolving  National  military  require¬ 
ments  dictate  similar  needs.  To  address 
these  needs  we  introduce  Cost  Exchange 
Ratios  as  metrics  of  both  cost  and  opera¬ 
tional  effectiveness  amenable  to  any  level 
of  MSA  from  a  simple  ad  hoc  spreadsheet 
simulation  (which  we  use  for  our  example,) 
through  formal,  detailed  analyses  such  as 
COEAs. 


II.  EFFECTIVENESS  ANALYSIS 
OVERVIEW 

Traditionally,  military  concepts,  whether 
they  be  new  weapon  systems,  new  organi¬ 
zations,  or  new  Tactics,  Techniques,  and 
Procedures,  have  been  analyzed  in  their 
natural  environment  of  military  operations. 
By  and  large,  these  efforts  have  been  con¬ 
ducted  using  simulations,  in  the  form  of 
either  field  exercises  and  trials,  or  machine 
based  constructs:  analytical;  man-in-the- 
loop;  or  a  combination.  Occasionally,  actual 
military  operations  permit  incorporation  of 
concepts  (e.g.,  the  Marines'  use  of  "Sticky 
Foam"  in  Somalia)  which  are  amenable  to 
post  hoc  analysis.  While  these  analyses  con¬ 
sider  various  metrics  such  as  force  move¬ 
ment  and  mission  accomplishment  as  well 
as  force  loss,  the  latter  remain  key  factors, 
even  for  non-weapon  concepts,  and  it  is 
thus  useful  to  define  exchange  ratios  in  this 
context. 

We  shall  limit  our  further  discussion  to 
constructive  simulation  although  the  elab¬ 
oration  to  other  forms  is  straightforward. 
Traditional  military  simulations,  be  they 
platform  or  aggregate  resolution,  are  two- 
sided  traditionally  designated  as  Red  (usu¬ 
ally  the  opponent  or  enemy)  and  Blue  (usu¬ 
ally  the  friendly  or  self).  In  platform  level 
simulation  (e.g.  CASTFOREM),  individual 
weapons  platform  types  (or  small  organi¬ 
zational  units — e.g.,  fire  teams)  are  the 
common  token  of  aggregation  with  the 
functional  operation  and  interaction  of  the 
platforms  typically  adjudicated  by  a  mix¬ 
ture  of  stochastic  sampling  and  rules  appli¬ 
cation  (which  may  itself  be  stochastic). 
While  several  aggregate  metrics  are  usually 
considered,  three  common  ones  are:  the  du¬ 
ration  (simulated  clock  time)  of  the  military 
operation  (r);  the  Killer-Victim  Scoreboard 
(KVS);  and  the  Rounds  Fired  Scoreboard 
(RFS). 

While  the  duration  seems  obvious  (but 
is  possibly  ambiguous),  the  other  two  are 
less  so.  Both  of  these  are  matrices.  As  we 
have  indicated,  it  is  common  in  these  sim¬ 
ulations  to  aggregate  in  terms  of  platform 
type  although  it  is  equally  feasible  to  ag¬ 
gregate  in  terms  of  military  organization. 
Let  us  designate  that  there  are  a  total  of  N 
platform  types  between  the  two  sides,  N  = 
Na  +  Nb,  where  NA  and  NB  are,  respec¬ 
tively,  the  number  of  Red  and  Blue  plat¬ 
form  types.  The  KVS  is  then  an  N  by  N 
matrix  whose  i,  i(i,  j  =  1  . . .  N)  entry  is  the 
number  of  the  i  type  platforms  killed  dur¬ 
ing  the  operation  by  the  fh  type  platforms. 
In  a  similar  manner,  the  RFS  is  also  an  N  by 
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N  matrix  whose  i,  j  entry  is  the  number  of 
rounds  fired  by  the  ith  type  platforms  at  the  fh 
type  platforms.  In  the  absence  of  fratricide, 
these  matrices  are  block  off  diagonal  with  zero 
diagonal  elements.  This  being  the  case,  we  may 
concentrate  on  these  non-zero  off-diagonal 
blocks  which  represent  the  number  of  Red,  Blue 
platforms  killed  (rounds  fired)  by  (at)  Blue,  Red 
platforms.  We  may  readily  designate  these  for 
mathematical  purposes  as  A AX  m  and  ABm  /,  l  = 
1 . . .  Na,  m  =  1 . . .  Nb  for  the  reduced  KVS  and 
A al/tn  and  A hml  for  the  reduced  RFS.  Initial  (start 
of  combat)  numbers  of  platforms  A,(0)  and 
Bm( 0)  are  parameters.  The  relationship  to  both 
homogeneous  and  heterogeneous  Lanchester 
attrition  theory  is  straightforward.  [Taylor, 
1983],  [Fowler,  1996] 


III.  EXCHANGE  RATIOS 


Exchange  ratios  have  a  long  history  of  us¬ 
age.  [Schenk,  1982]  Two  common  operational 
effectiveness  ratios  are  the  Loss  Exchange  Ra¬ 
tio,  defined  by. 


Ple(t)  = 


AB(t) 

A Mr)' 


2Nb  eBm  2Na  ABm  i(t) 

m=l  /= 1 

2N*efZN°  A A, Jr)' 

/=!  m= 1 


and  the  Force  (or  relative)  Exchange  Ratio,  de¬ 
fined  by, 

,  ,_A(0)  A(0)AB(t) 

Pfe(t)  -  B(Q)  Ple(V  ~  B(Q)  aa(t), 


1Na  ef  A/(0)  INb  el  2Na  AB„,  /(t) 

/=1  m= 1  /= 1 

~2Nb  eBmBm{ 0)EVr  AA;,m(r)' 

m= 1  l-l  m= 1 

(2) 

where  the  top  form  in  equations  (1)  and  (2)  is 
the  homogeneous  form  and  the  lower  is  the 
heterogeneous  form.  Homogeneous  and  heter¬ 
ogeneous  are  used  here  in  the  Lanchester  sense. 
The  coefficients  {e^)f  {ef}  are  the  aggregation 
coefficients  (or  scores)  for  the  Blue,  Red  sides, 
respectively,  normed  to  a  base  platform  type 
for  each,  and  t  is  the  duration  (simulated  clock 
time)  of  the  military  operation. 


The  Loss  Exchange  Ratio  is,  just  as  its  name 
implies,  the  ratio  of  Blue  combat  losses  to  Red 
combat  losses.  Values  of  this  ratio  less  than  one 
indicate  greater  Red  than  Blue  loss  and  thus 
may  indicate  superior  Blue  combat  effective¬ 
ness. 

This  comparison  tends  to  be  more  mean¬ 
ingful  if  the  two  forces  are  initially  approxi¬ 
mately  evenly  matched.  This  situation  has  not 
been  commonly  perceived  as  relevant,  first  dur¬ 
ing  Containment  when  Warsaw  Pact  forces  en¬ 
joyed  a  large  numeric  Order  of  Battle  superior¬ 
ity  over  NATO  forces,  and  now  under 
conditions  of  Force  Projection  from  CONUS. 
This  leads  to  consideration  of  the  Force  Ex¬ 
change  Ratio  which  is  just  the  ratio  of  the  frac¬ 
tional  Blue  losses  to  the  fractional  Red  losses. 
Values  of  this  ratio  less  than  one  indicate 
greater  fractional  Red  losses  than  fractional 
Blue  losses,  and  thus  indicate  superior  Blue 
combat  effectiveness. 

From  a  mathematical  standpoint,  both  of 
these  ratios  may  be  viewed  as  time  functions 
which  are  essentially  indeterminate  at  the  start 
of  combat  and  may  fluctuate  rapidly  until  both 
sides  incur  some  mathematically  reasonable 
number  of  losses.  Utility  of  these  ratios  presup¬ 
poses  their  stability  as  a  measure  of  combat 
evolution.  For  our  purposes  here,  we  assume 
this  stability  and  consider  these  ratios  as  fixed 
values  evaluated  at  the  end  of  the  combat.  Ex¬ 
tension  to  continuous  trajectories  over  most  of 
the  combat  duration  is  straightforward,  and 
while  valuable,  beyond  the  scope  of  this  brief 
paper. 

Both  of  these  ratios  are  concerned  only  with 
the  operational  effectiveness  of  the  two  forces 
without  consideration  of  their  cost  effective¬ 
ness.  Cost  effectiveness,  both  of  weapons  plat¬ 
forms  in  particular,  and  forces  in  general,  is  a 
matter  of  concern  both  in  terms  of  planning  and 
execution.  This  importance  is  demonstrated  by 
the  requirement  of  Cost  and  Operational  Effec¬ 
tiveness  Analyses  at  most  milestone  reviews 
during  the  life  cycle  of  weapon  acquisition  pro¬ 
grams  and  by  similar  analyses  associated  with 
force  and  operations  planning  used  in  develop¬ 
ing  campaign  plans,  national  and  regional  mil¬ 
itary  strategies,  and  budget  submissions. 

To  our  knowledge,  however,  the  common 
integration  of  cost  and  operational  effectiveness 
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is  at  "the  bottom  line."  This  has  the  advantage 
of  maintaining  independence  between  cost  and 
operational  analysis  efforts,  but  it  reduces  the 
flexibility  and  scope  of  analytical  efforts  to  "op¬ 
timize"  both  cost  and  operational  effectiveness. 
In  essence,  the  overall  "system'  of  cost  and 
operational  effectiveness  cannot  be  optimized 
although  the  "subsystems'  of  cost  effectiveness 
and  operational  effectiveness  can  be  optimized. 
This  may  result  in  overall  suboptimality  and 
either  exclude  possible  solutions  from  consid¬ 
eration  or  skew  the  decision  making  process. 


IV.  COST  EXCHANGE  RATIOS 

In  considering  the  costs  aspects  of  combat, 
we  may  identify  three  specific  areas: 

(1)  loss  of  friendly  assets; 

(2)  expenditure  of  ammunition  stockpile;  and 

(3)  differential  upkeep  of  forces; 

which  are  additional  costs  incurred  as  a  result 
of  the  combat.  The  first  of  these  is  the  most 
complicated.  At  the  tactical  level,  it  is  the  eco¬ 
nomic  loss  incurred  due  to  the  loss  of  weapons 
platforms  in  combat.  At  the  operational  level,  it 
could  also  include  future  losses  incurred  due  to 
these  present  losses.  At  the  strategic  level,  it 
could  further  include  reduction  of  the  nation's 
ability  to  prosecute  war  due  to  production  or 
morale  losses.  At  the  grand  strategic  level,  it 
could  even  include  long  term  national  losses 
resulting  from  territorial  losses  or  war  indem¬ 
nities.  The  scope  of  our  investigations  have 
thus  far  been  at  the  tactical  level  and  we  shall 
limit  our  further  discussion  to  this  level  while 
recognizing  both  greater  scope  and  further  op¬ 
portunity  to  explore  that  scope. 

The  second  cost  area  is  the  economic  loss 
incurred  by  engaging  (killing)  enemy  weapons 
platforms.  The  third  is  the  differential  cost  in¬ 
curred  by  deploying  the  force  to  the  combat 
locale  and  maintaining,  operating,  and  sup¬ 
porting  it  there.  Because  of  the  highly  complex 
nature  of  the  reasons  for  field  operations,  the 
accompanying  complexity  of  estimating  these 
differential  upkeep  costs  directly,  and  their  de¬ 
pendence  on  the  specifics  of  the  deployment, 
we  shall  incorporate  them  in  what  we  shall 
hereafter  refer  to  as  replacement  cost  for  this 


description,  recognizing  that  they  could  be  in¬ 
cluded  explicitly  in  a  detailed  study  using  this 
methodology. 

If  we  assume  that  under  ideal  conditions,  it 
is  desirable  to  maintain  a  prescribed  level  of 
force  capability,  represented  by  numbers  of 
weapons  platforms  and  ammunition  types, 
then  we  may  use  that  level  of  force  capability  as 
the  basis  of  definition  of  a  metric.  Particularly, 
if  that  level  is  to  be  maintained,  then  combat 
losses  must  be  replaced  after  combat  (ideally 
before  another  combat  occurs),  and  the  cost  of 
replacing  those  losses  provides  a  basis  for  a 
metric  of  the  cost  of  operational  effectiveness. 

The  concept  of  replacement  cost  is  some¬ 
what  different  from  the  normal  accounting  con¬ 
sideration  and  represents  some  simplification 
for  the  purpose  of  defining  a  metric.  While  the 
concept  is  readily  understandable  to  anyone 
who  has  lost  a  car  in  an  accident  or  has  replaced 
a  failed  household  appliance,  it  is  not  readily 
consistent  with  common  defense  accounting 
practice. 

Under  certain  circumstances,  replacement 
cost  may  actually  be  zero.  This  is  the  case  for 
weapons  developed  and  deployed  for  one  com¬ 
bat  (e.g.,  the  original  nuclear  weapons,)  or  for 
weapons  systems  (ammunitions)  which  are  at 
or  nearing  obsolescence.  For  the  former,  re¬ 
placement  was  never  intended  and  the  cost  is 
all  sunk.  For  the  latter,  the  old  platform  (am¬ 
munition)  would  have  been  replaced  (outside 
combat  conditions)  by  a  new  platform  (ammu¬ 
nition). 

If  we  neglect  these  two  cases,  then  we  may 
safely  assume  that  under  most  conditions,  re¬ 
placement  cost  is  nonzero,  but  it  is  not  obvi¬ 
ously  incremental.  In  particular,  we  must  note 
that  while  most  weapons  platforms  (and  am¬ 
munition)  are  produced  under  manufacturing 
conditions,  there  are  non-production  costs  as¬ 
sociated  with  technology  research  and  develop¬ 
ment,  product  development  and  testing,  and 
factory  tooling  and  start-up  costs.  These  costs 
must  already  have  been  paid  if  the  system  is 
fielded.  In  addition,  there  are  other  costs  which 
must  be  considered:  the  life  cycle  cost  of  main¬ 
taining  the  system;  and  the  cost  of  refactoriza¬ 
tion  and  production  if  initial  production  has 
been  completed. 
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These  are  all  factors  which  have  been  am¬ 
ply  considered  in  various  previous  cost  studies 
and  analyses,  so  we  shall  assume  suitable  ca¬ 
pability  for  developing  replacement  costs. 

Since  replacement  cost  is  thus  effectively 
the  cost  of  implementing  a  policy  of  inventory 
replenishment,  a  learning  curve  algorithm,  ad¬ 
justed  for  the  start-up  and  other  costs  men¬ 
tioned  above,  seems  appropriate.  This  is  a  tra¬ 
ditional  model  incorporating  production  rate 
variables  [Moses,  1995]  modified  to  incorporate 
a  constant  for  extraneous  variables.  The  incre¬ 
mental  unit  cost  model  [Sherrill]  for  learning 
curves  is  more  appropriate  than  the  cumulative 
average  cost  [Wright,  1936]  because  the  latter 
weights  early  instances  more  heavily  and  thus 
averages  out  period-to-period  changes  in  aver¬ 
age  cost.  [Liao,  1988]  This  model  has  the  math¬ 
ematical  form, 

I  =  aQbRd  +  K,  (3) 

where  I  is  the  unit  replacement  cost,  Q  is  the 
cumulative  quantity  produced,  R  is  the  unit 
production  rate,  a  is  the  first  unit  cost,  b  is  the 
learning  curve  slope,  d  is  the  production  rate 
slope,  and  K  is  the  additional  unique  unit  cost. 
The  latter  variable  may  be  expected  to  also 
include  logistics  and  maintenance  costs. 

This  model  may  also  be  used  to  describe 
the  cost  of  crew  replacement.  The  question  of 
the  value  of  a  human  being  is  charged  with 
emotional,  social,  and  cultural  overtones.  If  our 
consideration  is  limited  to  the  tactical  level, 
then  the  cost  of  crew  replacement  consists  of 
the  incremental  costs  of  recruiting,  training, 
and  deploying  the  crews  of  the  replacement 
systems  and  the  funeral  and  survivor  benefit 
costs  for  the  lost  crews,  all  of  which  may  be 
described  in  terms  of  a  production  model  such 
as  equation  (3).  At  the  operational  through 
grand  strategic  levels,  additional  costs  are  in¬ 
curred  which  have  increasing  subjective  nature. 

The  Loss  Exchange  Ratio,  a  meaningful 
metric  of  some  aspects  of  operational  effective¬ 
ness,  may  be  used  as  a  philosophical  spring¬ 
board  for  a  metric  of  cost  and  operational  ef¬ 
fectiveness.  We  may  recall  that  the  Loss 
Exchange  Ratio  is  the  ratio  of  the  fractional 
friendly  loss  to  the  fractional  enemy  loss.  For 
the  case  where  each  force  is  comprised  of  only 


one  platform  type  (homogeneity),  this  ratio 
may  be  formed  easily  and  directly.  When  one 
or  both  forces  consist  of  multiple  platform 
types  (or  equivalently,  organizational  types,) 
then  some  form  of  aggregation  is  necessary. 
Regardless,  the  Loss  Exchange  Ratio  is  funda¬ 
mentally  the  ratio  of  fractional  losses  at  a  level 
of  equivalence.  A  straightforward  extension  to 
include  cost  may  be  formed  by  defining  equiv¬ 
alence  in  terms  of  the  replacement  costs  of  the 
losses.  We  call  this  quantity  the  Cost  Exchange 
Ratio. 

As  we  have  indicated,  there  are  two  re¬ 
placement  costs  to  be  considered:  the  cost  of 
platform  replacement;  and  the  cost  of  ammuni¬ 
tion  replacement.  We  define  the  quantities  <pf, 
(p^  to  be  the  unit  platform  replacement  costs  for 
each  Red,  Blue  platform  type,  and  ^  to  be 
the  unit  ammunition  replacement  costs  for  each 
Red,  Blue  platform  type.  This  assumes  each 
component  has  only  one  weapon  system,  but 
this  is  not  a  restriction  except  as  a  simplifica¬ 
tion.  We  further  assume  the  unit  platform  re¬ 
placement  costs  to  include  the  cost  of  a  basic 
load  of  ammunition,  and  it  may  include  the 
platform's  share  of  the  ammunition  stockpile. 
With  these  definitions,  we  may  define  the  total 
force  replacement  costs  for  the  two  forces  as 

Na 

CA(t )  =  2 

1=1 

Nb 

CB(t )  -  2  <pBmBm(t).  (4) 

m= 1 

The  costs  to  each  force  due  to  their  combat 
losses  are  just  the  combination  of  platform  loss 
and  ammunition  loss, 

Na  Nb 

A  CA(t)  =  X2(<P?AA„,(f)  +  ^A(U0), 

l=lm=l 

Nb  Na 

A C„(t)  =  2  2(<f&ABm/f)  +  iffiA bm,m  (5) 

m=ll=l 

These  are  approximate  since  they  double  count 
the  rounds  fired  by  killed  platforms.  We  shall 
accept  this  discrepancy  as  a  necessary  restric¬ 
tion  since  it  is  essentially  impossible  to  calcu¬ 
late  in  an  aggregate  (e.g.,  Lanchester)  simula- 
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tion.  Alternately,  a  proportional  reduction  of 
the  RFS  using  the  ratios  of  surviving  to  total 
platforms  of  each  type  could  be  used  as  an 
approximation.  The  discrepancy  can  be  elimi¬ 
nated  by  modifying  a  platform  level  simulation 
to  produce  two  RFS,  one  for  rounds  fired  by 
killed  platforms  and  the  other  for  rounds  fired 
by  surviving  platforms.  The  latter  RFS  would 
then  be  used  in  equations  (5).  So  long  as  the 
ammunition  stockpiles  are  large,  the  ratio  of 
ammunition  (per  round)  cost  to  platform  cost  is 
small,  and  combat  losses  occur  for  all  plat¬ 
forms,  then  the  effects  of  this  discrepancy 
should  be  small. 

With  these  quantities,  we  may  now  define 
the  Cost  Exchange  Ratio  as. 


_  Q(0)  A CB(f) 

PceW  -  CB( 0)  A CA(ty 


(6) 


As  with  the  Force  Exchange  Ratio,  the  Cost 
Exchange  Ratio  is  just  the  (approximate)  ratio 
of  the  fractional  cost  loss  of  the  Blue  force  to  the 
fractional  cost  loss  of  the  Red  force.  Values  of 
this  ratio  less  than  one  indicate  that  the  Blue 
force  is  more  cost  effective  in  the  combat  than 
the  Red  force. 

This  presumes,  of  course,  that  the  economic 
bases  of  the  two  forces  (nations)  may  be  com¬ 
pared,  so  we  would  not  expect  this  metric  to  be 
useful  if  the  two  forces  represent  disparate  eco¬ 
nomic  cultures.  Further,  while  this  metric  does 
represent  a  combination  of  cost  and  operational 
effectiveness,  and  thus  adds  new  flexibility  and 
scope  to  analysis,  it  manifestly  cannot  represent 
a  complete  picture  of  combat.  We  shall  address 
this  consideration  specifically  below  with  the 
Normed  Cost  Exchange  Ratio. 

Several  limiting  behaviors  of  the  Cost  Ex¬ 
change  Ratio  may  be  noted.  For  the  homoge¬ 
neous  form  (whether  ab  initio  or  by  aggrega¬ 
tion),  the  ratio  reduces  to 


Pce(t) 


cpM(O)  9  +  pB(kill\shot )^(T) 

?B(0)-a7/  ,  ,  W 

9  AA(T)  +  p,(b7Z|s^)AB(T) 

(7) 


where  the  overbar  quantities  indicate  aggre¬ 
gates.  This  result  takes  advantage  of  the  inte¬ 
grated  Quadratic  Lanchester  Attrition  Differen¬ 
tial  Equation,  AA(t)  =  a/5B(f)d£',  and  a  ~  {rate  of 
fire)pB{kill\shot)/  to  form  Afr(r)  ~  /^(fa'ZZls/zof)"1 
AA(r). 

With  a  bit  of  algebra  and  the  use  of  equa¬ 
tion  (2),  equation  (7)  reduces  to 


PCE  — 


tffB  A(0) 

Pfe  +  - 

<PbVb  B(0) 

.  &A  S(0) 

1  +  “  PpE 

<PaPa  MO) 


(8) 


For  modem  weapon  systems  with  high  proba¬ 
bility  of  kill  given  a  shot,  platform  replacement 
costs  much  larger  than  ammunition  replace¬ 
ment  costs  (<p  »  if /),  and  approximately  equal 
initial  force  strengths,  equation  (8)  approaches 
the  form. 


Pce  —  Pfe /  (9) 

which  may  have  some  utility  as  a  rule  of  thumb 
or  decision  support  flag  for  more  detailed  in¬ 
spection  of  the  underlying  combat  results. 

The  problem  of  comparing  disparate  force 
(national)  economies  may  be  alleviated  by  mak¬ 
ing  use  of  an  aggregation  methodology  such  as 
Formal  Aggregation  Theory  [Fowler,  1997].  An 
aggregation  of  this  type  reduces  the  heteroge¬ 
neous  situation  to  a  homogeneous  one  in  terms 
of  a  base  platform  (or  unit)  type.  This  offers  an 
obvious  basis  of  cost  comparison  which  we  use 
in  the  Normed  Cost  Exchange  Ratio,  defined  as 


_  __  Ca(0)  ACa(t) 
Pc£  “  CA( 0)  ACa(t)' 


(10) 


where  C^(0)  and  ACa(t)  are  the  total  force  and 
combat  loss  replacement  costs  in  terms  of  a 
base  platform  type  aggregation.  These  latter 
represent  the  costs  of  an  equivalent  force  com¬ 
prised  only  of  the  base  platform  type.  This  ratio 
permits  comparisons  among  different  platform 
types  and  ammunitions  on  the  basis  of  equiva¬ 
lent  combats  within  a  common  economic 
framework. 


V.  EXAMPLE 

To  illustrate  these  ratios,  we  present  a  sim¬ 
ple  example  of  their  use.  We  make  use  of  the 
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Table  1.  Attrition  Rate  Coefficients  for  Example 


Target/ 

Blue  1 

Blue  2 

Target/ 

Red  1 

Red  2 

Firer 

Firer 

Red  1 

0.1 

0.2 

Blue  1 

0.15 

0.25 

Red  2 

0.3 

0.35 

Blue  2 

0.4 

0.3 

2X2  Intensive  Formal  Aggregation  of  the  het¬ 
erogeneous  to  homogeneous  Lanchester  prob¬ 
lem.  [Fowler,  1997]  The  attrition  rate  coeffi¬ 
cients  are  given  in  Table  1.  Rates  of  fire  for  all 
systems  are  four  rounds  per  time  unit.  The 
force  strength  trajectories  for  this  example,  both 
individual  and  aggregate,  are  shown  in  Figure 
1.  The  individual  force  strength  trajectories  are 
simple  (numerical)  integrations  of  the  heteroge¬ 
neous  Lanchester  attrition  differential  equa¬ 
tions  for  the  four  platform  types  (Red  1  and  2, 
Blue  1  and  2)  for  initial  force  strengths  of  100 
and  50  for  Red,  and  75  and  35  for  Blue.  The 
aggregate  force  strength  trajectories  (labeled 
Red  agg  and  Blue  agg)  are  the  homogeneous 
aggregates  (equivalent  Red,  Blue)  of  these  indi¬ 
vidual  trajectories.  The  aggregation  coeffi¬ 
cients,  which  are  derived  from  the  attrition  rate 
coefficients  of  Table  1,  are:  Red  (1.0,  0.95);  and 
Blue  (1.0, 1.40).  The  Red,  Blue  force  was  aggre¬ 


gated  using  Red,  Blue  1  platform  as  the  base,  as 
indicated  by  the  aggregation  coefficient  value 
of  1.  In  this  aggregation,  Red  2  platforms  have 
operational  "value'  95%  of  Red  1  platforms 
while  Blue  2  platforms  have  operational  value 
140%  of  Blue  1  platforms. 

The  costs  of  these  platforms  and  their  am¬ 
munition  are  given  in  Table  2.  The  Red  plat¬ 
forms  have  lower  cost  than  the  Blue  platforms, 
although  the  costs  of  the  ammunition  is  the 
same  for  platforms  labeled  1  and  2,  respec¬ 
tively.  These  values  roughly  represent  the  rel¬ 
ative  market  prices  of  Former  Soviet  Union  and 
NATO  ground  combat  platforms,  and  of  tank 
ammunition  and  second  generation  antitank 
guided  missiles. 

Loss  Exchange  (labeled  LER),  Force  Ex¬ 
change  (FER),  and  Cost  Exchange  (CER)  Ratios, 
and  Normed  Cost  Exchange  Ratios  for  the  Red 
and  Blue  forces  (NCERA,  NCERB)  are  shown  in 
Figure  2.  For  the  latter  two  ratios,  as  indicated 
above  by  the  aggregation  coefficients,  the  first 
system  type  of  each  force  is  the  base  system. 

From  Figure  2,  we  may  see  the  same  struc¬ 
ture  among  the  Loss,  Force,  and  Cost  Exchange 
Ratios.  Clearly,  Red  is  performing  to  greater 
effectiveness  than  Blue,  as  we  may  plainly  see 
by  examining  Figure  1.  Nonetheless,  Blue  is 
relatively  more  cost  effective  than  Red  as  dem- 
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Figure  1.  Force  Strength  Trajectories 
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Table  2.  Costs  for  Example  (dollars) 


Type/Cost 

Platform 

(millions) 

Ammunition 

(thousands) 

Red  1 

1 

1 

Red  2 

1 

10 

Blue  1 

4 

1 

Blue  2 

2 

10 

onstrated  by  the  smaller  magnitude  of  the  Cost 
Exchange  Ratio,  despite  the  fact  that  it  is 
greater  than  one.  This  apparent  contradiction  is 
a  direct  result  of  the  greater  Red  lethality 
against  Blue  2  and  its  relatively  lower  replace¬ 
ment  cost. 

The  two  Normed  Cost  Exchange  Ratios  re¬ 
late  costs  normed  to  the  first  platform  type  in 
each  case.  The  Red  Normed  Cost  Exchange  Ra¬ 
tio  is  greater  than  one,  which  indicates  decreas¬ 
ing  the  percentage  of  Red  2  would  be  more  cost 
effective.  This  result  can  be  seen  from  the  am¬ 
munition  replacement  cost  in  Table  2.  Similarly 
the  Blue  Normed  Cost  Exchange  Ratio  is  less 
than  one  indicating  that  an  increase  in  the  num¬ 
ber  of  platforms  of  the  second  type  with  a  de¬ 
crease  in  the  number  of  first  type  platforms 
would  be  more  cost  effective.  This  may  be  seen 


from  the  platform  replacement  cost  in  Table  2 
and  the  aggregation  coefficient  values  since  a 
system  that  is  half  as  expensive  and  40%  more 
deadly  will  be  more  cost  effective. 


VI.  CONCLUSION 

This  paper  presents  a  pair  of  ratios,  the 
Cost  Exchange  Ratio  and  the  Normed  Cost  Ex¬ 
change  Ratio,  which  are  metrics  that  combine 
cost  and  operational  effectiveness.  As  such  they 
extend  the  scope  and  flexibility  of  analysis  of 
weapon  system  acquisition,  force  structuring 
and  planning,  etc.  These  ratios  follow  directly 
from  the  definitions  of  replacement  cost  and 
Force  Exchange  Ratio. 

This  methodology  appears  to  offer  consid¬ 
erable  promise  as  an  analytical  tool.  For  exam¬ 
ple,  it  is  possible  (for  one  side,)  to  effectively  fix 
the  Force  Exchange  Ratio  and  minimize  cost  by 
adjusting  force  composition.  Alternately,  it  is 
also  possible  to  fix  cost  and  optimize  Force 
Exchange,  again  by  adjusting  force  composi¬ 
tion.  We  must  note  however,  that  since  each 
ratio  is  a  point  aggregation  for  a  specific  combat 
situation,  proper  consideration  should  be  given 
to  a  spectrum  of  combat  situations  as  a  frame¬ 
work  for  such  optimization. 


Time  (arbitrary  units) 


—•“LER 

CER  NCERB 

-■•■-FER 

-Ar’NCERA 

Figure  2.  Loss,  Force,  Cost,  and  Normed  Cost  Exchange  Ratios 
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Nonetheless,  given  such  consideration,  the 
Cost  Exchange  Ratio  could  be  used  to  support 
both  programmatic  and  force  structure  deci¬ 
sions.  This  is  particularly  important  in  a  period 
of  constrained  resources  and  uncertain  mission 
requirements.  It  provides  a  basis  of  consider¬ 
ation  of  system  modernization  or  improvement 
in  the  context  of  Cost  as  an  Independent  Vari¬ 
able.  Further,  within  a  system  of  systems  con¬ 
text,  either  for  a  single  service  or  jointly,  it 
provides  a  metric  for  constructing  a  balanced 
force  within  affordable  limits. 
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ABSTRACT 

his  paper  describes  an  analysis  effort 
initiated  by  the  Joint  Requirements 
Oversight  Council  (JROC)  of  the  Joint 
Chiefs  of  Staff  (JCS).  Decision  analysis  tech¬ 
niques,  to  include  Multi-attribute  Value 
Analysis  and  Cost-Benefit  Analysis,  were 
used  to  develop  a  methodology  for  the 
evaluation  of  alternate  reconnaissance  force 
mixes. 

Given  the  importance  of  joint  recon¬ 
naissance  to  today's  operational  command¬ 
ers  and  increasing  reliance  on  reconnais¬ 
sance  for  the  future,  the  JROC  recognized 
in  1995  that  it  had  no  means  to  make  force 
mix  decisions  in  terms  of  end-to-end  plat¬ 
form  capability  and  cost  across  all  compo¬ 
nents:  manned,  unmanned,  and  overhead. 
To  fulfill  this  need,  the  JROC  created  the 
Reconnaissance  Study  Group  (RSG)  and 
tasked  it  to  develop  and  implement  a  pro¬ 
cess  for  making  timely  and  informed  recon¬ 
naissance  force  mix  decisions. 

This  paper  describes  an  innovative 
methodology  for  determining  the  composi¬ 
tion  of  promising  reconnaissance  architec¬ 
tures  at  various  levels  of  investment  for  the 
2010  time  frame.  The  unique  aspects  of  the 
approach  are  its  broad  scope  and  scalability  in 
addressing  the  multiple  components  of  the 
architecture,  the  use  of  value  assessments 
based  upon  simulations  as  well  as  subjec¬ 
tive  expert  judgment  to  provide  traceability 
and  repeatability ,  and  its  treatment  of  cost  as 
an  independent  variable  (Rush,  1997)  in  the 
cost-benefit  analysis  of  future  force  mix  op¬ 
tions. 

BACKGROUND 

The  purpose  of  this  analysis  was  to 
develop  and  implement  a  process  for  mak¬ 
ing  timely  and  informed  reconnaissance 
force  mix  decisions  for  airborne  and 
spaced-based  platforms.  Given  the  impor¬ 
tance  of  joint  reconnaissance  support  to  to¬ 
day's  operational  commanders  and  increas¬ 
ing  future  reliance  on  reconnaissance,  the 
Joint  Requirements  Oversight  Council 
(JROC)  of  the  Joint  Chiefs  of  Staff  recog¬ 
nized  in  1995  that  it  had  no  means  to  make 
force  mix  decisions  in  terms  of  end-to-end 
platform  capability  and  cost  across  all  com¬ 
ponents:  manned,  unmanned,  and  over¬ 
head.  To  fulfill  this  need,  the  JROC  created 
the  Reconnaissance  Study  Group  (RSG), 
comprised  of  flag  officers  representing  the 


Services  and  key  agencies.  The  JROC 
tasked  the  RSG  to  develop  and  implement 
a  process  for  making  timely  and  informed 
reconnaissance  force  mix  decisions. 

The  JROC  requested  that  the  RSG  focus 
on  future  requirements  and  capabilities  in  a 
time  frame  no  earlier  than  2005.  The  JROC 
was  particularly  interested  in  the  effort  to 
define  criteria  and  metrics  for  force  struc¬ 
ture  discrimination  and  evaluation.  In  ac¬ 
cordance  with  the  JROC's  guidance,  the  fol¬ 
lowing  parameters  were  agreed  upon: 

•  The  scope  of  the  analysis  would  include 
unmanned,  manned,  and  overhead  re¬ 
connaissance  collection  platforms  as  well 
as  exploitation  and  dissemination  sys¬ 
tems. 

•  All  military  tasks  to  which  reconnaissance 
systems  contribute  would  be  considered, 
not  just  two  Major  Regional  Conflicts 
(MRCs). 

•  The  2010  time  frame  would  be  used  for 
both  requirements  and  available  capabil¬ 
ities  and  technologies. 

•  A  set  of  possible  reconnaissance  force 
mixes  would  be  developed  with  suffi¬ 
cient  breadth  to  enable  robust  insight 
into  possible  budget  cuts. 

•  Unlike  most  reconnaissance  studies,  this 
analysis  would  be  scenario-independent . 

•  The  analysis  would  have  an  operational 
focus  with  reconnaissance  requirements 
being  generated  by  users  (Unified  Com¬ 
mands,  Services,  and  Joint  Staff  repre¬ 
sentatives). 

•  Cost  and  benefit  to  the  user  would  both 
be  independent  variables  in  the  analysis; 
lower  cost  solutions  for  reconnaissance 
force  mixes  would  be  of  particular  inter¬ 
est. 

The  three  major  objectives  of  the  anal¬ 
ysis  were: 

•  Establish  an  initial  capability  to  assess 
force  mix  trades. 

•  Develop  a  decision  support  method  that 
provided  benefit  and  cost  comparisons  of 
reconnaissance  systems  in  terms  of  the 
overall  satisfaction  of  requirements . 

•  Provide  promising ,  candidate  force  mixes 
that  could  be  subjected  to  further  analy¬ 
sis  of  different  types,  including  detailed 
modeling  and  simulation  in  specific  sce¬ 
narios. 

Finally,  there  were  four  critical  issues 
in  the  design  of  the  analysis  that  had  to  be 
resolved:  (1)  how  should  the  architecture 
building  blocks  for  reconnaissance  force 
mixes  be  designed  so  that  both  creative  and 
exhaustive  sets  of  force  mixes  could  be  de- 
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fined  and  analyzed  in  a  reasonable  amount  of 
time?  (2)  how  should  modeling  be  used  to  ag¬ 
gregate  concepts  in  appropriate  places  in  order 
to  achieve  80%  of  the  desired  effect  (e.g.,  preci¬ 
sion,  accuracy)  with  only  20%  of  the  mathemat¬ 
ical  detail?  (3)  how  and  from  whom  should  the 
judgmental  inputs  be  obtained  so  that  they 
were  considered  valid?  and  (4)  how  should  the 
results  be  presented  so  that  they  are  meaning¬ 
ful  to  the  decision  makers? 

This  study  developed  an  analytic  process 
and  preliminary  analysis  recommendations. 
This  paper  focuses  on  the  analysis  process  be¬ 
cause  the  results  are  classified  and  were  not 
ends  in  themselves  but  have  been  fed  into  other 
decision  making  process.  The  detailed  results 
are  contained  in  "Final  Report  on  the  Recon¬ 
naissance  Study  Group  Cost-Benefit  Analysis 
of  Airborne  and  Space-Borne  Reconnaissance 
Force  Mixes"  (1996). 


METHODOLOGY  OVERVIEW 

The  analysis  process,  consisting  of  the  six 
steps  as  shown  in  Figure  1,  was  briefed  to  the 
JROC  prior  to  the  study  and  approved  by  them 
for  implementation.  The  RSG  monitored  the 
study  and  approved  the  study's  activities  at 
every  step.  The  six  steps  were: 

1.  Develop  and  prioritize  the  mission-driven 
tasks  and  intelligence  functions  (require¬ 
ments  criteria  tree  in  Figure  1), 


2.  Develop  "metrics"  for  evaluating  architec¬ 
tures  on  the  tasks, 

3.  Identify  reasonable  and  innovative  "archi¬ 
tectural  packages", 

4.  Evaluate  the  alternate  architectures  on  the 
metrics  (assess  capabilities), 

5.  Develop  a  cost  model  and  evaluate  costs  of 
the  architectures  (cost  data), 

6.  Perform  a  "cost-benefit  analysis"  of  the  al¬ 
ternate  architectures. 

The  analysis  process  integrates  subject  mat¬ 
ter  experts  (system  operators  and  information 
users),  quantitative  data,  qualitative  judgments, 
and  tools  into  a  structured  and  orderly  process 
to  provide  timely  and  meaningful  insight  and 
trends  to  senior  decision  makers  on  the  relative 
costs  and  benefits  of  reconnaissance  alternate 
architectures.  Each  step  is  described  in  more 
detail. 

Step  1 .  Develop  and  Prioritize  the 
Mission-driven  Tasks  and 
Intelligence  Functions 

We  began  by  considering  all  military  appli¬ 
cations  for  reconnaissance  that  were  found  in 
the  1995  National  Security  Strategy  and  the  1995 
National  Military  Strategy .  At  the  highest  level, 
there  are  four  support  elements:  political  infor¬ 
mational  military ,  and  economic.  The  focus  of 
this  study  was  on  military  effectiveness  includ- 
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ing  performance  issues,  other  factors,  and  risk. 
Performance  was  examined  in  three  categories: 
peacetime  engagement ,  deterrence  and  conflict  pre¬ 
vention,  and  fighting  and  winning  the  joint  con¬ 
flict.  Each  of  these  categories  was  broken  down 
further  into  tasks,  intelligence  functions,  and 
metrics.  During  this  first  step  the  emphasis  was 
placed  at  the  task  level. 

Figure  2  shows  this  hierarchy  down  to  the 
task  level.  The  numbers  reflect  the  relative  im¬ 
portance  of  improving  the  reconnaissance  per¬ 
formance  from  minimum  acceptable  to  desired 
in  each  task;  all  of  the  relative  swings  in  impor¬ 
tance  at  the  task  level  sum  to  100.  For  example, 
performance  was  given  80%  of  the  weight;  the 
80%  was  allocated  as  almost  10%  to  peacetime 
engagement,  almost  31%  to  deterrence  and  con¬ 
flict  prevention,  and  slightly  more  than  39%  to 
fight  and  win  the  joint  conflict.  The  allocation  of 
these  weights  to  the  next  lower  level  is  shown 
in  Figure  2.  The  decimal  points  are  an  artifact  of 
the  normalization  process  and  are  not  intended 
to  show  undo  precision  in  the  weighting  pro¬ 
cess,  which  was  the  consensus  judgment  of  rep¬ 
resentatives  from  each  CINC. 

The  category  of  "Other  Factors"  was  added 
to  capture  issues  that  were  specifically  part  of 
the  military  tasks,  intelligence  functions  and 


associated  metrics.  In  particular,  there  were 
three  categories  of  operational  factors  (opera¬ 
tional  constraint  such  as  overflight  of  other 
countries,  logistics  issues,  and  political  issues). 
Finally  risk  factors  were  identified  but  not  dis¬ 
aggregated  because  there  is  too  much  interac¬ 
tion  among  cost,  schedule  and  performance 
risk. 

To  achieve  agreement  on  the  structure 
shown  in  Figure  2  and  elicit  the  weights,  a 
three-day  decision  conference  was  held  with  rep¬ 
resentatives  from  the  following  organizations: 

•  CINCs  (ACOM,  CENTCOM,  PACOM,  SO- 
COM,  SOUTHCOM,  STRATCOM,  SPACE- 
COM); 

•  Services  (Army,  Navy,  Marine  Corps,  Air 
Force); 

•  Joint  Staff  (J2/J8); 

•  Agencies  (NSA,  CIO,  NRO,  DARO,  DIA, 
CISA,  DMA,  BMDO); 

•  Program  Offices. 

Decision  conferences  (see  Watson  and 
Buede,  1987;  Buede  and  Bresnick,  1992)  are  in¬ 
tensive,  facilitated  working  sessions  that  use 
quantitative  models  to  capture  qualitative  judg¬ 
ments.  The  weights  were  assessed  from  the 
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"bottom-up"  using  an  indirect  assessment  tech¬ 
nique  called  the  balance  beam  (Watson  and 
Buede,  1987).  This  technique  uses  conjoint  anal¬ 
ysis  to  establish  relationships  between  the 
weights  of  groups  factors  and  then  infer  nu¬ 
merical  weights  from  the  relationships.  The 
participants  were  asked  to  agree  on  the  rela¬ 
tionships  through  a  period  of  discussion  and 
voting  where  needed  to  resolve  disagreements. 
The  weights  reflect  three  considerations:  (1) 
how  important  will  the  requirement  be  to  the 
nation  in  2010;  (2)  what  is  the  probability  of 
occurrence  of  the  requirement  in  2010;  and  (3) 
how  relevant  are  reconnaissance  systems  to  ac¬ 
complishing  the  tasks  and  intelligence  func¬ 
tions?  Thus,  a  very  important  requirement 
could  receive  a  low  weight  in  this  analysis  if 
reconnaissance  had  little  to  contribute. 


Step  2.  Develop  Metrics  for 
Evaluating  Architectures  on  the 
Tasks  and  Intelligence  Functions 

For  each  performance  task,  key  intelligence 
functions  were  identified  to  include  intelligence 
preparation  of  the  battlefield ,  targeting ,  battle  dam¬ 
age  assessment ,  indications  and  warning ,  situation 
awareness ,  and  mapping/ char  ting! geodesy.  These 
intelligence  functions  were  defined  on  the  basis 
of  research  ("Assured  Support  to  Operational 
Commanders",  1994;  and  "Future  National  In¬ 
telligence  Review",  1992)  and  consultation  with 
the  participants  of  the  second  decision  confer¬ 
ence,  who  were  largely  the  same  as  the  partic¬ 
ipants  of  the  first  decision  conference. 

Relevant  metrics  for  each  intelligence  func¬ 
tion  were  selected  from  the  set  of  metrics 
shown  in  Table  1. 

A  second  three-day  decision  conference 
was  held  in  which  a  "strawman"  set  of  intelli¬ 
gence  functions  and  metrics  for  each  bottom 
level  entry  in  Figure  2  was  presented  to  the 
participants  for  review  and  revision.  Before  an 
evaluation  of  the  architectures  was  performed, 
each  metric  was  defined  in  detail  to  ensure  that 
the  participants  had  a  common  understanding 
of  how  system  capability  would  be  assessed. 

One  of  the  key  analytical  challenges  was 
"how  much  modeling  is  enough?"  Clearly,  we 
could  have  evaluated  every  architecture  on  ev¬ 
ery  metric  of  every  reconnaissance  function  for 
every  task  and  intelligence  function — a  massive 
task.  By  recognizing  that  some  tasks  were  much 


Table  1.  Value  Metrics 


METRIC 

METRIC 

TYPE 

Quality 

Level  of  Detail 

Geolocation  Accuracy 

Timeliness 

Timeliness 

Responsiveness 

Update  Frequency 

Quantity 

Area  Coverage  (IMINT) 

Point  Targets  (IMINT) 

Erlangs  (SIGINT) 

Signals  of  Interest  (SIGINT) 

more  heavily  weighted  than  others,  we  devel¬ 
oped  an  "economy  of  modeling"  rule  of  thumb 
that  reduced  the  modeling  load  with  little  or  no 
impact  on  results.  We  categorized  the  tasks  as 
shown  in  Table  2. 

For  highly  weighted  tasks  such  as  counter¬ 
ing  weapons  of  mass  destruction,  the  analysis 
included  all  relevant  intelligence  functions  and 
all  relevant  metrics  for  each;  for  moderately 
weighted  tasks  such  as  wartime  power  projec¬ 
tion,  the  analysis  only  included  the  single  most 
relevant  intelligence  function  and  all  relevant 
metrics  for  the  function;  and  for  the  lowest 
weighted  tasks  such  as  force  generation,  the 
analysis  restricted  attention  to  the  single  most 
relevant  intelligence  function  and  the  single, 
most  pertinent  metric  for  that  function. 

After  the  metrics  were  defined,  value 
curves  (Watson  and  Buede,  1987)  were  needed 
to  measure  user  satisfaction  as  a  function  of 
system  capability.  Figure  3  presents  an  example 
of  a  notional  performance  value  curve  for  time¬ 
liness.  Each  curve  relates  user  satisfaction  (ex¬ 
pressed  on  a  0  to  100  point  scale)  to  the  perfor¬ 
mance  of  a  group  of  reconnaissance  systems  on 
tasks  and  intelligence  functions  for  a  specific 
metric.  The  RSG  analysis  used  available  intelli¬ 
gence  community-generated  value  curves 
(such  as  from  the  Future  National  Intelligence 
Review ,  1992)  as  well  as  expert  judgment.  These 
curves  reflect  the  notion  of  decreasing-returns- 
to-scale  common  in  economics  and  decision 
analysis. 

For  each  metric,  a  family  of  curves  was 
developed  rather  than  a  single  curve.  Each 
curve  in  a  family  had  the  same  shape  and  was 
defined  by  three  points:  the  maximum  satisfac¬ 
tion  point  that  was  assigned  a  score  of  100,  the 
minimum  satisfaction  point  that  was  assigned  a 
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Table  2.  Categorization  of  Military  Performance  Tasks  by  Weight 


CATEGORY  OF  WEIGHT  VALUE 

Low 

(below  1.5) 

Moderate 

(between  1.5  and  7.4) 

High 

(above  7.4) 

Arms  Control 

Nuclear  Deterrence 

Fight  &  Win  (2 

Sanctions  Enforcement 

Crisis  Response 

MRC  focus) 

Counter  Weapons  of 

PERFORMANCE 

Science  &  Technology 
Peacekeeping 

Humanitarian  Operations 

Win  the  Information  War 
Counterdrug  & 
Counterterrorism 
Noncombatant  Evacuation 

Mass  Destruction 

TASK 

Nation  Assistance 

Win  the  Peace 

Regional  Alliances 

Security  Assistance 

Military  to  Military  Contacts 
Confidence  Building  Measures 
Force  Generation 

Operations 

Peace  Enforcement 

Wartime  Power  Projection 

Notional  Utility  Curve  on  Timeliness 


Figure  3.  Notional  Value  Curve  for  Timeliness 


score  of  10,  and  the  nominal  required  capability 
that  was  assigned  a  score  of  70.  These  scores 
were  used  in  the  Future  National  Intelligence 
Review  (1992);  the  participants  agreed  that  this 
structure  was  acceptable.  The  curves  had  slight 
decreasing  returns  to  scale.  Note,  any  capability 
below  the  minimum  satisfaction  point  was 
given  a  score  of  0.  This  family  of  curves  recog¬ 
nized  that  the  same  capability  might  be  valued 
differently  in  different  tasks.  For  example,  a 
timeliness  capability  of  several  hours  may  get 
100  value  points  in  a  peacetime  engagement 


context,  but  in  a  wartime  context,  timeliness  of 
seconds  or  minutes  may  be  required  to  get  100 
points. 

Value  curves  were  also  developed  for  the 
"Other  Factors"  criteria,  including  Risk. 

During  the  second  decision  conference  the 
participants  reviewed  the  family  of  curves 
available  for  each  bottom  level  task,  intelligence 
function,  and  metric  and  selected  the  curve 
judged  to  be  most  relevant.  In  some  cases  the 
three  points  defining  the  value  curve  were 
changed  to  better  reflect  the  judgment  of  the 
CINC,  Service,  and  Joint  Staff  representatives. 
This  consensus  was  usually  obtained  by  open 
discussion  followed  by  general  agreement.  In 
cases  where  there  was  insufficient  agreement,  a 
vote  was  taken.  These  participants  also  pro¬ 
vided  weights  across  the  intelligence  functions 
and  value  curves  using  the  same  process  as  was 
employed  during  the  first  decision  conference. 
It  is  important  to  note  that  the  value  curves 
were  developed  independent  of  the  alternatives 
and  should  remain  valid  as  new  alternatives 
are  introduced  in  the  future. 

With  the  addition  of  performance  metrics 
and  notional  value  curves  for  each  reconnais¬ 
sance  function,  the  performance  evaluation  hi¬ 
erarchy  would  be  as  shown  in  Figure  4  for  Fight 
and  Win  the  Joint  Conflict.  Similar  trees  were 
developed  for  the  other  elements  of  Perfor¬ 
mance  for  the  Other  Factors. 
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The  tree  structure  now  has  two  main 
branches  for  Military  Support:  performance 
and  other  factors.  Within  the  performance 
branch  there  are  four  levels:  level  of  conflict 
(peacetime  engagement,  deterrence  and  conflict 
prevention,  and  fight  and  win),  conflict  tasks 
(e.g.,  nation  assistance  and  wartime  power  pro¬ 
jection),  intelligence  functions  (e.g.,  targeting 
and  situation  awareness),  and  metrics.  For  the 
other  factors  branch,  there  are  two  factors:  op¬ 
erational  and  risk.  Operational  factors  is  di¬ 
vided  into  issues  of  operations,  logistics,  and 
politics.  Every  metric  and  the  four  bottom  level 
branches  under  other  factors  had  a  value  curve 
defined  for  it. 


Step  3.  Identify  Alternative 
Architectural  “Levels” 

An  architecture  refers  to  a  system  of  sys¬ 
tems;  in  this  particular  case,  to  a  system  of 
airborne  and  space-borne  reconnaissance  plat¬ 
forms,  each  platform  being  comprised  of  one  or 
more  systems.  For  the  purposes  of  this  study 
the  reconnaissance  platforms  were  grouped 
into  six  distinct  components: 


•  Manned  Multi-Intelligence  (Multi-INT)  plat¬ 
forms 

•  Manned  Signals  Intelligence  (SIGINT)  plat¬ 
forms 

•  Manned  Imagery  Intelligence  (IMINT)  plat¬ 
forms 

•  Endurance  Unmanned  Aerial  Vehicles 
(UAV)  platforms 

•  Tactical  Unmanned  Aerial  Vehicles  plat¬ 
forms 

•  Overhead  satellites  platforms 

Figure  5  illustrates  how  the  reconnaissance 
components  were  configured  for  this  analysis 
to  represent  an  overall  system  of  systems  archi¬ 
tecture.  For  each  component  (row),  the  lettered 
blocks  represent  alternate  packages  (referred  to 
as  levels)  for  the  row.  An  architecture  is  defined 
by  selecting  one  package  (level)  for  each  row. 

Each  component  package  depicted  in  Fig¬ 
ure  5  included  "end-to-end"  systems  (including 
processing,  exploitation,  and  dissemination 
systems)  as  well  as  the  front-end  system  re¬ 
quired  to  task  reconnaissance  systems  and  de¬ 
liver  "products"  to  users.  Each  of  the  packages, 
i.e.,  each  cell  in  Figure  5,  is  itself  a  combination  of 
specific  reconnaissance  platforms  in  varying 
configurations  and  quantities.  Each  platform 
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An  architecture  =  one  package  from  each  component  (e.g.,  J+F+D+H+E+B) 
Figure  5.  Architectural  Components 


contains  the  requisite  systems  needed  to  com¬ 
plete  its  missions.  For  example,  the  eight 
manned  SIGINT  packages  shown  in  Figure  5 
might  be  made  up  of  the  notional  platforms 
shown  in  Table  3.  Here,  Level  A  is  made  up 
only  of  System  F  aircraft  and  hence  is  a  low 
cost,  bare  bones  option.  Conversely,  Level  H  is 
made  up  of  sizable  quantities  from  all  seven 
platform  types,  most  of  which  have  upgraded 
sensors  and  re-engining,  and  hence  is  a  highly 
capable  option  with  associated  higher  life  cycle 
cost. 

A  third  3-day  decision  conference  was  held 
to  identify  the  platforms  and  associated  sys- 


Table  3.  Notional  Packages  of  the  Manned 
SIGINT  Component:  Comprised  of  Platform 
Quantities  and  Configurations 


SYSTEM 

TYPES 

PACKAGES 

A 

B 

C 

D 

E 

F 

G 

H 

Platform  A 

0 

0 

3* 

3 

3* 

3 

0 

3* 

Platform  B 

0 

8 

8* 

16 

8* 

16 

16* 

16* 

Platform  C 

0 

14 

24* 

30 

24* 

42 

42 

42* 

Platform  D 

0 

0 

0 

8 

16 

16 

16*+ 

16*+ 

Platform  E 

0 

12 

12 

6 

0 

12 

12* 

12* 

Platform  F 

2 

2 

2 

2 

2* 

2 

2*+ 

2*+ 

Platform  G 

0 

3 

3 

3 

3 

3 

3*+ 

3*+ 

*  improved  sensors 
+  re-engining 


terns  that  made  up  each  component  and  the 
numbers  and  capabilities  of  each  platform  in 
each  package  of  the  components.  Each  cell  of 
each  row  in  Table  3  then  had  to  be  evaluated  in 
terms  of  its  costs  and  its  benefits  on  all  of  the 
value  curves. 

Step  4.  Evaluate  the  Architectures 
on  the  Metrics 

The  initial  effort  of  evaluating  the  user  sat¬ 
isfaction  of  each  package  in  each  component 
used  a  linear-additive  scoring  rule  for  combin¬ 
ing  weights  and  scores  from  the  value  curves. 
This  approach  assumed  preferential  indepen¬ 
dence  of  the  evaluation  criteria  at  every  level  of 
the  evaluation  hierarchy  (Keeney  and  Raiffa, 
1976),  which  proved  to  be  too  general  an  as¬ 
sumption  for  the  reconnaissance  functions  and 
performance  metrics.  To  address  specific  inter¬ 
actions  between  platforms,  metrics  and  intelli¬ 
gence  functions,  a  set  of  scoring  rules  and  as¬ 
sumptions  were  developed  that  allowed  for 
combining  metrics  for  each  intelligence  func¬ 
tion  in  a  nonadditive  way  based  upon  the  dy¬ 
namic  and  operational  realities  of  how  the  sys¬ 
tems  would  be  employed  (e.g.,  weather 
considerations,  resource  contention,  simulta¬ 
neous  tasking,  etc.).  These  scoring  rules  are 
summarized  in  the  Appendix.  Each  assumption 
and  rule  was  carefully  documented  and  fully 
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supports  the  degree  of  traceability  and  repeat¬ 
ability  of  the  decision  methodology  desired  by 
the  JROC. 

The  scoring  "rollup"  at  the  level  of  intelli¬ 
gence  function  and  above  assumed  a  linear- 
additive  scoring  rule.  This  assumption  was  vet¬ 
ted  with  the  participants  of  the  study  and 
judged  to  be  an  appropriate  assumption  at  this 
level  of  the  hierarchy.  The  rollup  on  the  perfor¬ 
mance  hierarchy  was  done  in  an  extensive  set 
of  interactive  spreadsheets.  Thus,  it  was 
straightforward  to  use  the  scoring  methodol¬ 
ogy  and  the  spreadsheet  file  to  examine  a  broad 
range  of  available  simulation  results  and  differ¬ 
ent  assumptions  on  system  capability.  This 
technique  also  guaranteed  consistency  and  re¬ 
peatability  of  the  results.  The  output  of  the 
scoring  process  was  a  single  benefit  score  for 
each  level  of  each  architectural  component  for 
each  of  the  three  performance  areas:  peacetime 
engagement,  deterrence  and  conflict  prevention ,  and 
fight  and  win  the  joint  war. 

Scores  for  the  "Other  Factors"  criteria  were 
assessed  directly  from  the  value  curves  devel¬ 
oped  for  those  criteria  for  each  package  of  each 
component.  For  example,  each  package  was 
scored  on  risk.  Packages  with  only  existing 
platforms  were  scored  highest.  Packages  with 
platforms  for  which  program  offices  were  not 
yet  defined  were  scored  lowest.  The  weighted 
average  for  "Other  Factors"  assumed  a  linear- 
additive  scoring  rule,  which  was  also  judged  to 
be  appropriate  for  these  criteria  by  the  partici¬ 
pants  of  the  study. 

To  summarize  the  equation  for  computing 
the  benefit  for  each  package  in  each  component 
of  a  possible  architecture  was: 


B  Wpl  ^ 2- ^wCiwTijWjFijkS 
\  i  j  k 

+  Wof(wopf(woSo  +  WlSl  +  WpSp)  +  WrSr) 


where 


wp  =  weight  for  Performance 
wc i  =  weight  for  Conflict  category 
i 

wTi.  —  weight  for  Task  j  in  Conflict 
category  i 

z^ipijk  =  weight  for  Intelligence 
Function  k  in  Task  j  in 
Conflict  category  i 
w0 F  =  weight  for  Other  Factors 
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weight  for  Operational 
Factors 

weight  for  Operational 
Issues 

weight  for  Logistics  Issues 
weight  for  Political  Issues 
weight  for  Risk  Factor 
Value  Score  on  Intelligence 
Factor  k  of  Conflict  I  and 
Task  j 

Value  Score  on  Operational 
Issues 

Value  Score  on  Logistics 
Issues 

Value  Score  on  Political 
Issues 

Value  Score  on  Risk 


Step  5.  Develop  a  Cost  Model  and 
Evaluate  Costs  of  the  Architectures 

The  major  objective  of  the  cost  analysis  was 
to  capture  "end-to-end"  reconnaissance  system 
Life  Cycle  Costs  (LCCs)  from  1997  through 
2020.  For  this  analysis,  the  end-to-end  use  of  a 
system  was  defined  as  the  systems  (ground  and 
airborne /overhead),  maintenance,  and  man¬ 
power  required  to  direct  the  system's  data  col¬ 
lection  activity,  execute  the  data  collection,  dis¬ 
seminate  the  raw  data,  and  analyze  and 
redirect  the  data  collected  by  the  system.  To 
accomplish  this  goal,  system  development,  pro¬ 
curement,  and  operations  and  support  (O&S) 
costs  for  each  platform  configuration  (including 
upgrades)  were  computed  and  aggregated 
based  upon  the  number  of  platforms.  In  addi¬ 
tion,  the  costs  of  the  associated  ground  stations 
and  additional  airborne  assets  needed  to  exe¬ 
cute  the  mission  were  computed  and  propor¬ 
tionally  added  to  the  platform  costs. 

A  second  critical  goal  of  this  effort  was  to 
develop  a  comprehensive  and  traceable  recon¬ 
naissance  LCC  development  approach  that 
would  provide  consistent  and  comparable  LCC 
estimates  for  each  level  in  the  RSG  cost-benefit 
analysis.  The  approach  provided  visibility  into 
all  the  cost  estimating  parameters,  such  as  first 
unit  costs,  learning  curve  effects,  costs  per  fly¬ 
ing  hour,  and  pre-planned  product  improve¬ 
ment  (P3I)  costs,  and  global  parameters,  such  as 
prior  quantity,  procurement  quantity,  average 
annual  flying  hours,  etc.  The  approach  pro¬ 
vided  the  ability  to  perform  "what-if"  excur- 
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sions  around  these  parameters  for  sensitivity 
analysis  and  the  capability  to  add  or  mix  con¬ 
figurations. 

The  key  challenge  was  to  obtain  consensus 
on  these  end-to-end  LCC  positions  from  repre¬ 
sentatives  associated  with  each  system  in¬ 
cluded  in  the  analysis.  A  one-day  decision  con¬ 
ference  was  held  on  costing  issues.  Based  upon 
data  provided  or  sanctioned  by  these  represen¬ 
tatives,  reasonable  LCC  positions  were  devel¬ 
oped  that  attained  this  consensus.  The  confer¬ 
ence  was  followed  by  an  open  review  of  all  of 
the  LCC  positions  by  the  representatives  from 
each  system.  Disconnects  and  disagreements 
were  discussed  and  new  LCC  positions  were 
developed  as  needed.  The  output  of  the  costing 
process  was  a  single  LCC  for  each  package  of 
each  architectural  component. 

Step  6.  Perform  a  Cost-Benefit 
Analysis  of  the  Architectures 

Once  a  cost  and  a  benefit  measure  was 
determined  for  each  level  of  each  architectural 
component,  a  cost-benefit  analysis  using  a  Pa¬ 
reto-optimal,  efficient  frontier  approach  was 
completed  to  determine  the  best  allocation  of 
resources  across  all  components  (Buede  and 
Bresnick  1992).  Commercial  off-the-shelf  soft¬ 
ware  called  EQUITY  was  used  to  develop  a  "1" 
to  "n"  list  that  achieves  the  most  bang  for  the 
buck  at  increasing  LCC  points.  See  Watson  and 
Buede  (1987)  for  a  discussion  of  this  approach 
and  Kirkwood  (1997)  for  a  more  general  discus¬ 
sion  of  this  topic.  The  major  assumption  made 


here  is  that  both  the  cost  and  benefits  associated 
with  a  given  component  were  independent  of 
the  packages  chosen  on  other  components. 

An  alternate  architecture  is  defined  by  se¬ 
lecting  one  package  from  each  architectural 
component.  The  small  circle  in  the  center  of  the 
football  in  Figures  6a  and  6b  represents  a  single 
architecture.  For  any  budget  target,  it  is  clear  in 
Figure  6b  that  the  most  benefit  is  achieved  if 
there  is  an  architecture  along  the  top  of  the 
football  that  is  near  the  budget  amount.  If  the 
dot  architecture  in  Figure  6b  has  a  LCC  of  C, 
then  architecture  X,  which  also  costs  C,  gets 
more  "bang-for-the-buck".  There  is  also  an  ar¬ 
chitecture  Y  that  gets  the  same  benefit  as  the 
dot  architecture  but  is  significantly  cheaper. 
There  are  more  than  310,000  possible  architec¬ 
tures  based  upon  the  14,  8,  12,  11,  7  and  3 
packages  in  the  six  components,  see  Figure  5. 
These  288,000  plus  architectures  have  the 
"nice"  mathematical  property  of  falling  into  a 
football  shaped  region  in  the  LCC  and  benefit 
space,  called  the  Pareto-optimal  space  as  shown 
in  Figure  6a.  For  any  proposed  architecture  that 
falls  within  the  football  shaped  region,  there  is 
a  better  (X)  and  cheaper  (Y)  way  to  spend  the 
same  amount  of  resources  as  shown  in  Fig¬ 
ure  6b. 

This  analytical  approach  focuses  on  finding 
the  architectures  along  the  top  of  the  football. 
This  "efficient  frontier"  establishes  the  best  or- 
der-of-buy  of  packages  in  the  architectural 
components;  this  order-of-buy  is  called  the  con¬ 
vex  hull  in  mathematical  programming.  It  is 
calculated  by  taking  the  incremental  benefit  in 
going  from  one  package  to  the  next  in  a  row 


Figure  6a.  A  Single  Architecture  (the  small  circle).  The  Pareto-Optimal  Space,  and  the  Efficient  Frontier 

Figure  6b.  Better  and  Cheaper  Solutions 
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and  dividing  by  the  incremental  cost,  then  or¬ 
dering  the  ratios  from  best  to  worst.  This  "mar¬ 
ginal"  analysis  provides  far  more  insight  into 
achieving  efficiencies  than  simply  calculating 
total  benefit/cost  ratios.  There  were  54  different 
optimal  increments  out  of  the  more  than 
288,000  possible  architectures.  On  the  basis  of 
this  optimal  order  we  plotted  the  total  LCC  and 
benefit  of  each  of  the  54  optimal  architectures 
on  the  convex  hull  of  the  Pareto-optimal,  or 
efficient,  frontier.  Decision  makers  refer  to  it  as 
the  top  of  the  football.  Note  this  analysis  pro¬ 
cess  did  not  compute  all  of  the  Pareto  optimal 
architectures  that  fell  between  each  pair  of 
points  on  the  convex  hull.  The  purpose  of  this 
analysis  was  not  to  determine  the  optimal  way 
to  spend  a  pre-defined  amount  of  money,  but 
rather  to  provide  insight  into  the  sets  of  pack¬ 
ages  that  were  near  the  optimal  allocation  of 
life  cycle  cost  during  the  timeframe  of  the 
study. 

A  final  3-day  decision  conference  was  held, 
and  using  the  benefits  and  costs  assessed  ear¬ 
lier,  we  developed  the  "football"  for  the  recon¬ 
naissance  mix  as  shown  in  Figure  7. 

Since  the  conclusions  of  the  study  have  not 
yet  been  finalized  at  the  JROC  level,  no  specific 
results  will  be  discussed  here.  However,  it  is 
clear  that  the  methodology  presents  a  straight¬ 
forward  way  to  investigate  architectures  that 
are  near  the  "knee  of  the  curve",  thus  achieving 
a  very  efficient  use  of  resources.  By  varying  the 


weights  on  the  evaluation  factors,  we  per¬ 
formed  extensive  sensitivity  analyses  and  de¬ 
veloped  various  footballs  for  different  world¬ 
wide  scenarios. 


CONCLUSIONS 

The  combined  multi-attribute  value,  Pa- 
reto-optimality  approach,  implemented  in  a  de¬ 
cision  conferencing  framework  is  a  viable 
methodology  for  end-to-end  analysis  of  mili¬ 
tary  systems.  It  reveals  alternatives  that  may 
offer  substantial  economies  without  significant 
loss  in  benefit,  as  well  as  alternatives  that  get 
more  benefit  for  the  same  level  of  expenditure. 
The  approach  can  be  used  to  achieve  consensus 
by  bringing  together  operators  (CINC  and  Ser¬ 
vice  representatives),  requirements  managers, 
and  systems  developers. 

The  Background  section  stated  that  there 
were  four  issues  that  needed  to  be  resolved  as 
this  analysis  proceeded. 

•  How  should  the  architecture  building  blocks 
for  reconnaissance  force  mixes  be  designed  so 
that  both  creative  and  exhaustive  sets  of 
force  mixes  could  be  defined  and  analyzed  in 
a  reasonable  amount  of  time?  This  issue  was 
resolved  by  breaking  the  total  set  of  recon¬ 
naissance  resources  into  the  six  components 
defined  in  Figure  5.  At  the  end  of  the  analysis 


Figure  7.  Cost-Benefit  "Football"  for  reconnaissance  Mixes 
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the  study  team  had  a  long  debate  about 
whether  this  division  could  have  been  im¬ 
proved.  There  were  arguments  for  and 
against  defining  many  more  components  al¬ 
though  everyone  agreed  that  it  would  have 
been  a  big  mistake  to  have  allowed  each 
platform  to  be  its  own  component;  the  logic 
for  this  conclusion  was  that  there  was  too 
much  redundant  capability  among  many  of 
the  platforms  and  this  detailed  division 
would  have  greatly  over  valued  the  architec¬ 
ture.  Since  the  main  issue  with  this  analysis 
was  that  the  architecture  as  a  whole  seemed 
to  be  overvalued  (see  Step  4,  Figure  7,  and 
the  appendix),  there  was  a  strong  argument 
for  even  fewer  components  than  the  six  that 
were  defined.  However,  even  after  the  dis¬ 
cussion  the  group  was  unable  to  develop  a 
different  set  of  components  that  everyone 
agreed  would  have  been  better.  This  was  a 
key  to  a  successful  and  valid  analysis. 

•  How  should  modeling  be  used  to  aggregate 
concepts  in  appropriate  places  in  order  to 
achieve  80%  of  the  desired  effect  (e.g.,  preci¬ 
sion ,  accuracy )  with  only  20%  of  the  math¬ 
ematical  detail?  Here  the  study  group  felt 
that  the  decisions  to  limit  the  intelligence 
functions  and  metrics  for  conflict  tasks  with 
low  and  moderate  weights  were  valid  aggre¬ 
gations  and  did  not  detract  in  any  way  from 
the  final  results.  In  addition,  there  were  key 
uncertainties  that  were  embedded  in  the 
judgmental  inputs.  More  explicit  representa¬ 
tion  of  these  uncertainties  would  have  been 
very  valuable. 

•  How  and  from  whom  should  the  judgmental 
inputs  be  obtained  so  that  they  were  consid¬ 
ered  valid?  The  process  of  convening  mili¬ 
tary  and  civilian  representatives  from  the 
key  military  commands  and  government 
agencies  worked  very  well.  These  represen¬ 
tatives  were  knowledgeable  and  were  will¬ 
ing  to  rise  above  the  parochial  interests  of 
their  own  organization  and  address  the  over¬ 
all  value  to  military  support.  In  addition, 
these  representatives  were  able  to  maintain 
continuity  across  all  of  the  conferences  so 
that  valuable  time  did  not  have  to  be  devoted 
to  revisiting  old  issues.  The  consensus  of 
these  participants  was  that  this  analysis  pro¬ 
cess  was  an  important  and  valuable  ap¬ 
proach  to  examining  force  mix  decisions. 

•  How  should  the  results  be  presented  so  that 
they  are  meaningful  to  the  decision  makers? 
The  football  analogy  and  the  order-of-buy 


listing  proved  to  be  useful  to  communicating 
the  results  to  flag  officers  with  little  time  to 
dig  into  the  analysis  details.  The  concept  of 
breaking  the  reconnaissance  architecture  into 
six  components,  with  each  component  hav¬ 
ing  several  platforms  was  easy  to  under¬ 
stand. 

The  strengths  of  the  method  are  that: 

•  It  works — it  is  scaleable  in  time  and  dollars; 
it  isolates  the  key  and  viable  decision  alter¬ 
natives  (i.e.,  it  gets  to  the  "80%  solution" 
quickly);  it  captures  the  impact  of  cost  and 
military  benefit;  and  it  produces  traceable 
rationale. 

•  It  builds  consensus  among  participants — it 

accommodates  both  expert  judgment  and 
quantitative  performance  data;  it  addresses 
the  "soft"  factors  in  decision  making;  it  uses 
both  a  top-down  approach  to  achieve  a  struc¬ 
tured,  broad  view  and  a  bottoms-up  ap¬ 
proach  to  get  the  necessary  levels  of  detail;  it 
facilitates  sensitivity  analysis  on  key  param¬ 
eters;  and  it  is  easy  to  re-execute  using  dif¬ 
ferent  data  sets  or  assumptions. 

The  methodology  also  has  its  limitations: 

•  The  results  are  no  better  than  the  quality  of 
people  and  data  involved  in  the  process; 

•  It  can  require  a  large  number  of  judgments; 

•  It  imposes  some  assumptions  of  benefit  and 
cost  independence  among  the  rows  that  may 
not  be  true  in  the  real  world;  and 

•  It  can  take  a  lot  of  time  and  effort. 

As  with  most  operations  research  prob¬ 
lems,  there  were  the  analytical  challenges  that 
we  had  to  face,  but  there  were  also  the  "real- 
world"  challenges  that  were  even  more  diffi¬ 
cult.  These  included: 

•  How  do  we  get  many  organizations  that  are 
competing  for  resources ,  particularly  organi¬ 
zations  that  have  not  always  worked  well 
together ,  to  put  parochialism  aside  and 
think  “ corporately "? — the  decision  confer¬ 
ence  provided  the  forum  for  all  organiza¬ 
tions  to  be  heard  and  to  present  their  posi¬ 
tions,  to  exchange  information  openly,  and 
to  work  towards  informed  consensus.  Addi¬ 
tionally,  using  a  quantitative  framework 
took  much  of  the  emotionalism  out  of  the 
debate.  Bringing  the  right  people  to  the  table 
is  always  a  key  to  success.  We  were  fortunate 
to  have  senior  level,  forward  thinkers  who 
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did,  in  fact,  defend  their  own  positions  yet 
were  willing  to  think  "jointly"  as  well. 

•  How  do  we  deal  with  a  massive  number  of 
issues ,  judgments,  and  measurements  within 
the  limited  timeframe  and  scope  of  the  anal¬ 
ysis? — the  key  here  was  "economy  of  mod¬ 
eling".  By  applying  the  same  Pareto  tech¬ 
niques  used  in  the  approach  to  our  analytical 
efforts,  we  could  get  to  the  80%  solution 
quickly,  and  then  decide  where  it  was  worth 
our  effort  to  model  more  extensively.  We 
constantly  had  to  fight  the  urge  to  "get  more 
data",  and  we  had  to  remain  on  a  course  to 
reach  a  recommendation  before  the  problem 
had  changed. 

•  How  do  you  keep  people  from  believing  too 
much  in  the  computerized  numerical  re¬ 
sults? — Once  people  get  involved  with  the 
inertia  of  the  modeling  activity,  it  is  too  easy 
to  accept  results  that  come  out  of  the  back 
end.  Each  step  of  the  way,  we  had  to  apply  a 
"gut  check"  to  the  results,  and  if  they  didn't 
pass,  we  had  to  determine  why.  This  neces¬ 
sitated  revisiting  assumptions,  making  scor¬ 
ing  rules  more  specific,  and  developing  in¬ 
novative  modeling  approaches  for  areas  that 
had  not  been  previously  modeled.  Flexibility 
and  adaptability  were  key  ingredients  along 
with  a  well-documented  audit  trail  of  ratio¬ 
nale  which  is  often  more  important  than  the 
numerical  results  themselves. 

On  balance,  this  effort  allowed  us  to  merge 

people,  process,  and  data  to: 

•  Provide  insight  that  is  both  timely  and  mean¬ 
ingful  to  senior  decision  makers  as  to  the 
relative  worth  of  competing  military  support 
tasks  and  intelligence  functions; 

•  Validate  a  methodology  for  a  broad  and 
complex  problem;  and 

•  Illuminate  policy  issues  and  relationships  be¬ 
tween  specific  systems,  costs,  and  levels  of 
user  satisfaction. 

We  believe  that  it  is  a  viable  methodology 

for  future  efforts. 


Acronyms 

BMDO — Ballistic  Missile  Defense  Office 
CIO — Central  Imagery  Office 
DARO — Defense  Airborne  Reconnaissance  Of¬ 
fice 


DIA — Defense  Intelligence  Agency 
DMA — Defense  Mapping  Agency 
IMINT — Imagery  Intelligence 
IPB — Intelligence  Preparation  of  the  Battlefield 
ISR — intelligence,  surveillance,  and  reconnais¬ 
sance 

JCS— Joint  Chiefs  of  Staff 

JROC — Joint  Requirements  Oversight  Council 

LCC — Life  Cycle  Cost 

MRC — Major  Regional  Conflict 

Multi-INT — multi-source  intelligence 

NRO — National  Reconnaissance  Office 

NSA — National  Security  Agency 

O&S — operations  and  support 

P3I — pre-planned  product  improvement 

RSG — Reconnaissance  Study  Group 

SIGINT — Signals  Intelligence 

UAV — Unmanned  Aerial  Vehicles 
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Appendix 

The  scoring  process  developed  eight  scor¬ 
ing  rules  to  account  for  the  complexities  of 
scoring  a  package  vice  a  single  platform.  This 
was  complicated  by  the  fact  that  all  platforms 
did  not  have  a  one-to-one  correlation  in  benefit 
and  therefore  the  benefit  of  each  platform  could 
not  be  simply  aggregated.  The  following  rules 
accounted  for  these  complexities: 

1.  Weighted  Average  Rule:  When  appropriate, 
an  individual  platform's  contribution  to  a 
package's  score  for  Quality  and  Timeliness 
metrics,  was  weighted  based  upon  the  plat¬ 
form's  quantity  performance.  If  a  platform 
scored  high  on  a  Level  of  Detail  curve  but 
contributed  only  25%  of  the  Area  Coverage, 
its  contribution  to  the  overall  Level  of  Detail 


score  was  reduced  to  25%  of  what  it  would 
have  been.  This  precluded  package  scores 
from  being  overinflated  due  to  unrealistic 
package  performance. 

2.  Residual  Value  Rule:  If  a  platform  did  not 
contribute  at  least  the  Residual  Value  (10%) 
of  the  Required  Value,  then  its  contribution 
to  that  package  was  not  counted  in  the  ben¬ 
efit  score.  In  other  words,  if  a  platform 
scored  high  in  Geolocation  Accuracy  but 
contributed  less  than  the  specified  Residual 
Value  in  Area  Coverage,  then  its  contribu¬ 
tion  toward  that  package  was  not  included. 
This  precluded  the  contribution  to  overall 
package  score  from  a  platform  that  stood  no 
likelihood  of  ever  being  used. 

3.  Nuclear  Deterrence  Rule:  The  overall  air¬ 
borne  contribution  in  the  Nuclear  Deter¬ 
rence  military  performance  task  was  de¬ 
creased  to  10%  and  the  overhead 
contribution  was  decreased  to  20%.  This  was 
done  to  capture  the  unavailability  of  a  plat¬ 
form  to  collect  necessary  information. 

4.  Targeting  Rule:  Most  SIGINT  platforms  had 
minimum  contribution  to  scores  when  the 
Targeting  (TGT)  intelligence  function  was 
selected.  This  was  done  to  preclude  contri¬ 
bution  to  overall  package  score  by  platforms 
that  normally  would  not  be  used  to  support 
Targeting. 

5.  BDA/MC&G  Rule:  For  current  platform  ca¬ 
pability  and  CONOPS,  SIGINT  contribution 
to  the  Battle  Damage  Assessment  (BDA)  in¬ 
telligence  function  was  decreased  to  10%. 
Also,  for  the  one  case  where  the  Mapping, 
Charting,  and  Geodesy  (MC&G)  metric  was 
selected,  SIGINT  was  not  scored.  This  was 
done  to  preclude  contribution  to  overall 
package  score  by  platforms  that  normally 
would  not  be  used  to  support  BDA  or  per¬ 
form  MC&G  missions. 

6.  Multi-INT  Rule:  The  Multi-INT  platforms 
were  scored  if  they  were  able  to  achieve  the 
Residual  Value  or  greater  by  either  its 
SIGINT  or  IMINT  capability,  or  both.  If  the 
Multi-INT  platform  could  not  meet  the 
SIGINT  requirements  because  of  the  Resid¬ 
ual  Value  Rule  (see  Rule  #2,  above),  yet  was 
able  to  meet  or  exceed  the  IMINT  require¬ 
ments,  then  IMINT  credit  was  accounted  for 
in  the  scoring.  This  provided  appropriate 
credit  to  the  multi-functional  platforms  for 
being  able  to  use  whatever  sensors  that  were 
on  board  to  satisfy  the  requirements  of  a 
given  mission. 
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7.  Allocation  Rule:  A  platform  quantity  contri¬ 
bution  was  based  upon  the  weights  of  the 
intelligence  function.  If  a  collector  could  col¬ 
lect  140  Point  Targets  in  support  of  a  military 
task,  and  the  intelligence  functions  selected 
for  that  task  were  weighted  30%  IPB,  40% 
TGT,  and  30%  BDA,  then  those  140  Point 
Targets  were  distributed  to  those  tasks  pro¬ 
portionately.  (Namely,  42  Point  Targets  to 
IPB,  56  Point  Targets  to  TGT,  and  42  Point 
Targets  to  BDA).  This  compensated  for  the 


dynamic  that  in  a  performance  task  where 
multiple  intelligence  functions  are  being 
scored,  it  was  infeasible  that  platforms 
would  be  tasked  in  a  sequential  manner, 
rather  they  would  be  utilized  nearly  simul¬ 
taneously. 

8.  Exclusion  Rule:  If  a  platform  would  never 
contribute  to  a  task,  then  it  was  not  scored. 
This  precluded  a  platform  from  scoring  in  a 
task  where  it  never  would  be  used. 
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Computing  Advances  in  Military  OR 

Advanced  Computing 

Advanced  Distributed  Systems  (DIS) 

Cost  Analysis 

Wargaming 

Length  of  Papers 

Submissions  will  normally  range  from  10-30  pages  (double  spaced,  12  pitch,  including  illustra¬ 
tions).  Exceptions  will  be  made  for  applications  articles  submitted  with  a  senior  decision-maker  letter 
signed  by  the  Secretary  of  Defense. 
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TITLE  OF  SECTIONS 
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Title  of  Subsections 
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Title  Subsection  of  a  Subsection 

If  required  subsections  sections  may  be  divided  into  subsections.  Each  subsection  title  will  be 
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Paper  Electronic  Submission  with  Figures,  Graphs  and  Charts 

After  the  article  is  accepted  for  publication,  an  electronic  version  of  the  manuscript  must  be 
submitted  in  Microsoft  Word  or  WordPerfect.  For  each  figure,  graph,  and  chart,  please  include  a 
camera-ready  copy  on  a  separate  page.  The  figures,  graphs,  and  tables  should  be  of  sufficient  size  for 
the  reproduced  letters  and  numbers  to  be  legible.  Each  illustration  must  have  a  caption  and  a 
number. 

Mathematical  and  Symbolic  Expressions 

Authors  should  put  mathematical  and  symbolic  expressions  in  Microsoft  Word  or  WordPerfect 
equations.  Lengthy  expressions  should  be  avoided. 

Footnotes 

We  do  not  use  footnotes.  Parenthetical  material  may  be  incorporated  into  a  notes  section  at  the 
end  of  the  text,  before  the  acknowledgment  and  references  sections.  Notes  are  designated  by  a 
superscript  letter  at  the  end  of  the  sentence. 
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